{"thread":{"id":"59937","subject":"[PATCH v4] unit tests: Add a project plan document","startedAt":"2023-06-30T22:51:32Z","lastAt":"2023-11-09T23:15:11Z","messageCount":67,"participants":["Josh Steadmon","Junio C Hamano","Phillip Wood","phillip.wood123@gmail.com","Oswald Buddenhagen","Christian Couder"],"isPatch":true,"patchVersion":4,"patchTotal":null},"messages":[{"id":"479075","messageId":"0169ce6fb9ccafc089b74ae406db0d1a8ff8ac65.1688165272.git.steadmon@google.com","threadId":"59937","inReplyTo":"20230517-unit-tests-v2-v2-0-8c1b50f75811@google.com","subject":"[PATCH v4] unit tests: Add a project plan document","fromName":"Josh Steadmon","fromEmail":"steadmon@google.com","sentAt":"2023-06-30T22:51:19Z","receivedAt":"2023-06-30T22:51:32Z","isPatch":true,"sender":{"key":"steadmon@google.com","avatar":"https://avatars.githubusercontent.com/u/2654920?v=4"},"body":"In our current testing environment, we spend a significant amount of\neffort crafting end-to-end tests for error conditions that could easily\nbe captured by unit tests (or we simply forgo some hard-to-setup and\nrare error conditions). Describe what we hope to accomplish by\nimplementing unit tests, and explain some open questions and milestones.\nDiscuss desired features for test frameworks/harnesses, and provide a\npreliminary comparison of several different frameworks.\n\nCoauthored-by: Calvin Wan \nSigned-off-by: Calvin Wan \nSigned-off-by: Josh Steadmon \n---\nUnit tests additionally provide stability to the codebase and can\nsimplify debugging through isolation. Turning parts of Git into\nlibraries[1] gives us the ability to run unit tests on the libraries and\nto write unit tests in C. Writing unit tests in pure C, rather than with\nour current shell/test-tool helper setup, simplifies test setup,\nsimplifies passing data around (no shell-isms required), and reduces\ntesting runtime by not spawning a separate process for every test\ninvocation.\n\nThis patch adds a project document describing our goals for adding unit\ntests, as well as a discussion of features needed from prospective test\nframeworks or harnesses. It also includes a WIP comparison of various\nproposed frameworks. Later iterations of this series will probably\ninclude a sample unit test and Makefile integration once we've settled\non a framework. A rendered preview of this doc can be found at [2].\n\nIn addition to reviewing the document itself, reviewers can help this\nseries progress by helping to fill in the framework comparison table.\n\n[1] https://lore.kernel.org/git/CAJoAoZ=Cig_kLocxKGax31sU7Xe4==BGzC__Bg2_pr7krNq6MA@mail.gmail.com/\n[2] https://github.com/steadmon/git/blob/unit-tests-asciidoc/Documentation/technical/unit-tests.adoc\n\nTODOs remaining:\n- List rough priorities across comparison dimensions\n- Group dimensions into sensible categories\n- Discuss pre-existing harnesses for the current test suite\n- Discuss harness vs. framework features, particularly for parallelism\n- Figure out how to evaluate frameworks on additional OSes such as *BSD\n and NonStop\n- Add more discussion about desired features (particularly mocking)\n- Add dimension for test timing\n- Evaluate remaining missing comparison table entries\n\nChanges in v4:\n- Add link anchors for the framework comparison dimensions\n- Explain \"Partial\" results for each dimension\n- Use consistent dimension names in the section headers and comparison\n tables\n- Add \"Project KLOC\", \"Adoption\", and \"Inline tests\" dimensions\n- Fill in a few of the missing entries in the comparison table\n\nChanges in v3:\n- Expand the doc with discussion of desired features and a WIP\n comparison.\n- Drop all implementation patches until a framework is selected.\n- Link to v2: https://lore.kernel.org/r/20230517-unit-tests-v2-v2-0-21b5b60f4b32@google.com\n\n Documentation/Makefile | 1 +\n Documentation/technical/unit-tests.txt | 196 +++++++++++++++++++++++++\n 2 files changed, 197 insertions(+)\n create mode 100644 Documentation/technical/unit-tests.txt\n\ndiff --git a/Documentation/Makefile b/Documentation/Makefile\nindex b629176d7d..3f2383a12c 100644\n--- a/Documentation/Makefile\n+++ b/Documentation/Makefile\n@@ -122,6 +122,7 @@ TECH_DOCS += technical/scalar\n TECH_DOCS += technical/send-pack-pipeline\n TECH_DOCS += technical/shallow\n TECH_DOCS += technical/trivial-merge\n+TECH_DOCS += technical/unit-tests\n SP_ARTICLES += $(TECH_DOCS)\n SP_ARTICLES += technical/api-index\n \ndiff --git a/Documentation/technical/unit-tests.txt b/Documentation/technical/unit-tests.txt\nnew file mode 100644\nindex 0000000000..e302a0e40f\n--- /dev/null\n+++ b/Documentation/technical/unit-tests.txt\n@@ -0,0 +1,196 @@\n+= Unit Testing\n+\n+In our current testing environment, we spend a significant amount of effort\n+crafting end-to-end tests for error conditions that could easily be captured by\n+unit tests (or we simply forgo some hard-to-setup and rare error conditions).\n+Unit tests additionally provide stability to the codebase and can simplify\n+debugging through isolation. Writing unit tests in pure C, rather than with our\n+current shell/test-tool helper setup, simplifies test setup, simplifies passing\n+data around (no shell-isms required), and reduces testing runtime by not\n+spawning a separate process for every test invocation.\n+\n+We believe that a large body of unit tests, living alongside the existing test\n+suite, will improve code quality for the Git project.\n+\n+== Definitions\n+\n+For the purposes of this document, we'll use *test framework* to refer to\n+projects that support writing test cases and running tests within the context\n+of a single executable. *Test harness* will refer to projects that manage\n+running multiple executables (each of which may contain multiple test cases) and\n+aggregating their results.\n+\n+In reality, these terms are not strictly defined, and many of the projects\n+discussed below contain features from both categories.\n+\n+\n+== Choosing a framework & harness\n+\n+=== Desired features\n+\n+[[tap-support]]\n+==== TAP support\n+\n+The https://testanything.org/[Test Anything Protocol] is a text-based interface\n+that allows tests to communicate with a test harness. It is already used by\n+Git's integration test suite. Supporting TAP output is a mandatory feature for\n+any prospective test framework.\n+\n+In the comparison table below, \"True\" means this is natively supported.\n+\"Partial\" means TAP output must be generated by post-processing the native\n+output.\n+\n+Frameworks that do not have at least Partial support will not be evaluated\n+further.\n+\n+[[diagnostic-output]]\n+==== Diagnostic output\n+\n+When a test case fails, the framework must generate enough diagnostic output to\n+help developers find the appropriate test case in source code in order to debug\n+the failure.\n+\n+[[parallel-execution]]\n+==== Parallel execution\n+\n+Ideally, we will build up a significant collection of unit test cases, most\n+likely split across multiple executables. It will be necessary to run these\n+tests in parallel to enable fast develop-test-debug cycles.\n+\n+In the comparison table below, \"True\" means that individual test cases within a\n+single test executable can be run in parallel. \"Partial\" means that test cases\n+are run serially within a single executable, but multiple test executables can\n+be run at once (with proper harness support).\n+\n+[[vendorable-or-ubiquitous]]\n+==== Vendorable or ubiquitous\n+\n+If possible, we want to avoid forcing Git developers to install new tools just\n+to run unit tests. So any prospective frameworks and harnesses must either be\n+vendorable (meaning, we can copy their source directly into Git's repository),\n+or so ubiquitous that it is reasonable to expect that most developers will have\n+the tools installed already.\n+\n+[[maintainable-extensible]]\n+==== Maintainable / extensible\n+\n+It is unlikely that any pre-existing project perfectly fits our needs, so any\n+project we select will need to be actively maintained and open to accepting\n+changes. Alternatively, assuming we are vendoring the source into our repo, it\n+must be simple enough that Git developers can feel comfortable making changes as\n+needed to our version.\n+\n+In the comparison table below, \"True\" means that the framework seems to have\n+active developers, that it is simple enough that Git developers can make changes\n+to it, and that the project seems open to accepting external contributions (or\n+that it is vendorable). \"Partial\" means that at least one of the above\n+conditions holds.\n+\n+[[major-platform-support]]\n+==== Major platform support\n+\n+At a bare minimum, unit-testing must work on Linux, MacOS, and Windows.\n+\n+In the comparison table below, \"True\" means that it works on all three major\n+platforms with no issues. \"Partial\" means that there may be annoyances on one or\n+more platforms, but it is still usable in principle.\n+\n+[[lazy-test-planning]]\n+==== Lazy test planning\n+\n+TAP supports the notion of _test plans_, which communicate which test cases are\n+expected to run, or which tests actually ran. This allows test harnesses to\n+detect if the TAP output has been truncated, or if some tests were skipped due\n+to errors or bugs.\n+\n+The test framework should handle creating plans at runtime, rather than\n+requiring test developers to manually create plans, which leads to both human-\n+and merge-errors.\n+\n+[[runtime-skippable-tests]]\n+==== Runtime-skippable tests\n+\n+Test authors may wish to skip certain test cases based on runtime circumstances,\n+so the framework should support this.\n+\n+[[scheduling-re-running]]\n+==== Scheduling / re-running\n+\n+The test harness scheduling should be configurable so that e.g. developers can\n+choose to run slow tests first, or to run only tests that failed in a previous\n+run.\n+\n+\"True\" means that the framework supports both features, \"Partial\" means it\n+supports only one (assuming proper harness support).\n+\n+[[mock-support]]\n+==== Mock support\n+\n+Unit test authors may wish to test code that interacts with objects that may be\n+inconvenient to handle in a test (e.g. interacting with a network service).\n+Mocking allows test authors to provide a fake implementation of these objects\n+for more convenient tests.\n+\n+[[signal-error-handling]]\n+==== Signal & error handling\n+\n+The test framework must fail gracefully when test cases are themselves buggy or\n+when they are interrupted by signals during runtime.\n+\n+[[coverage-reports]]\n+==== Coverage reports\n+\n+It may be convenient to generate coverage reports when running unit tests\n+(although it may be possible to accomplish this regardless of test framework /\n+harness support).\n+\n+[[project-kloc]]\n+==== Project KLOC\n+\n+WIP: The size of the project, in thousands of lines of code. All else being\n+equal, we probably prefer a project with fewer LOC.\n+\n+[[adoption]]\n+==== Adoption\n+\n+WIP: we prefer a more widely-used project. We'll need to figure out the best way\n+to measure this.\n+\n+[[inline-tests]]\n+==== Inline tests\n+\n+Can the tests live alongside production code in the same source files? This can\n+be a useful reminder for developers to add new tests, and keep existing ones\n+synced with new changes.\n+\n+=== Comparison\n+\n+[format=\"csv\",options=\"header\"]\n+|=====\n+Framework,\"<>\",\"<>\",\"<>\",\"<>\",\"<>\",\"<>\",\"<>\",\"<>\",\"<>\",\"<>\",\"<>\",\"<>\",\"<>\",\"<>\",\"<>\"\n+https://lore.kernel.org/git/c902a166-98ce-afba-93f2-ea6027557176@gmail.com/[Custom Git impl.],[lime-background]#True#,[lime-background]#True#,?,[lime-background]#True#,[lime-background]#True#,[lime-background]#True#,[lime-background]#True#,?,?,[red-background]#False#,?,?,?,?,?\n+https://cmocka.org/[cmocka],[lime-background]#True#,[lime-background]#True#,?,[red-background]#False#,[yellow-background]#Partial#,[yellow-background]#Partial#,?,?,?,[lime-background]#True#,?,?,?,?,?\n+https://libcheck.github.io/check/[Check],[lime-background]#True#,[lime-background]#True#,?,[red-background]#False#,[yellow-background]#Partial#,[lime-background]#True#,?,?,?,[red-background]#False#,?,?,?,?,?\n+https://github.com/rra/c-tap-harness/[C TAP],[lime-background]#True#,[red-background]#False#,?,[lime-background]#True#,[yellow-background]#Partial#,[yellow-background]#Partial#,?,?,?,[red-background]#False#,?,?,?,?,?\n+https://github.com/silentbicycle/greatest[Greatest],[yellow-background]#Partial#,?,?,[lime-background]#True#,[yellow-background]#Partial#,?,?,?,?,[red-background]#False#,?,?,?,?,?\n+https://github.com/Snaipe/Criterion[Criterion],[lime-background]#True#,?,?,[red-background]#False#,?,[lime-background]#True#,?,?,?,[red-background]#False#,?,?,?,?,?\n+https://github.com/zorgnax/libtap[libtap],[lime-background]#True#,?,?,?,?,?,?,?,?,?,?,?,?,?,?\n+https://www.kindahl.net/mytap/doc/index.html[MyTAP],[lime-background]#True#,?,?,?,?,?,?,?,?,?,?,?,?,?,?\n+https://nemequ.github.io/munit/[µnit],[red-background]#False#,-,-,-,-,-,-,-,-,-,-,-,-,-,-\n+https://github.com/google/cmockery[cmockery],[red-background]#False#,-,-,-,-,-,-,-,-,-,-,-,-,-,-\n+https://github.com/lpabon/cmockery2[cmockery2],[red-background]#False#,-,-,-,-,-,-,-,-,-,-,-,-,-,-\n+https://github.com/ThrowTheSwitch/Unity[Unity],[red-background]#False#,-,-,-,-,-,-,-,-,-,-,-,-,-,-\n+https://github.com/siu/minunit[minunit],[red-background]#False#,-,-,-,-,-,-,-,-,-,-,-,-,-,-\n+https://cunit.sourceforge.net/[CUnit],[red-background]#False#,-,-,-,-,-,-,-,-,-,-,-,-,-,-\n+|=====\n+\n+== Milestones\n+\n+* Settle on final framework\n+* Add useful tests of library-like code\n+* Integrate with Makefile\n+* Integrate with CI\n+* Integrate with\n+ https://lore.kernel.org/git/20230502211454.1673000-1-calvinwan@google.com/[stdlib\n+ work]\n+* Run alongside regular `make test` target\n\nbase-commit: a9e066fa63149291a55f383cfa113d8bdbdaa6b3\n-- \n2.41.0.255.g8b1d071c50-goog\n\n"},{"id":"479076","messageId":"xmqq8rc05unt.fsf@gitster.g","threadId":"59937","inReplyTo":"0169ce6fb9ccafc089b74ae406db0d1a8ff8ac65.1688165272.git.steadmon@google.com","subject":"Re: [PATCH v4] unit tests: Add a project plan document","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2023-07-01T00:42:46Z","receivedAt":"2023-07-01T00:42:56Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Josh Steadmon writes:\n\nI'll normalize this one to match prevailing use.\n\n> Coauthored-by: Calvin Wan \n\n$ git log --since=6.months --pretty=raw --no-merges |\n sed -n -e 's/^ \\([^ :]*-by:\\).*/\\1/p' |\n sort | uniq -c | sort -n | sed -e '/^ *1 /d'\n 5 Tested-by:\n 15 Suggested-by:\n 24 Co-authored-by:\n 30 Reported-by:\n 34 Reviewed-by:\n 38 Helped-by:\n 68 Acked-by:\n 1786 Signed-off-by:\n\n> Signed-off-by: Calvin Wan \n> Signed-off-by: Josh Steadmon \n> ---\n\nThanks.\n"},{"id":"479077","messageId":"xmqq4jmo5tpl.fsf@gitster.g","threadId":"59937","inReplyTo":"0169ce6fb9ccafc089b74ae406db0d1a8ff8ac65.1688165272.git.steadmon@google.com","subject":"Re: [PATCH v4] unit tests: Add a project plan document","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2023-07-01T01:03:18Z","receivedAt":"2023-07-01T01:03:30Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Josh Steadmon writes:\n\n> In our current testing environment, we spend a significant amount of\n> effort crafting end-to-end tests for error conditions that could easily\n> be captured by unit tests (or we simply forgo some hard-to-setup and\n> rare error conditions). Describe what we hope to accomplish by\n> implementing unit tests, and explain some open questions and milestones.\n> Discuss desired features for test frameworks/harnesses, and provide a\n> preliminary comparison of several different frameworks.\n>\n> Coauthored-by: Calvin Wan \n> Signed-off-by: Calvin Wan \n> Signed-off-by: Josh Steadmon \n> ---\n\n> TODOs remaining:\n> - List rough priorities across comparison dimensions\n> - Group dimensions into sensible categories\n> - Discuss pre-existing harnesses for the current test suite\n> - Discuss harness vs. framework features, particularly for parallelism\n> - Figure out how to evaluate frameworks on additional OSes such as *BSD\n> and NonStop\n> - Add more discussion about desired features (particularly mocking)\n> - Add dimension for test timing\n> - Evaluate remaining missing comparison table entries\n\nListing these explicitly here is very much appreciated. Thanks.\n\n"},{"id":"480272","messageId":"c7dca1a805a16fd4fd68e86efeec97510e3ac4b8.1691449216.git.steadmon@google.com","threadId":"59937","inReplyTo":"0169ce6fb9ccafc089b74ae406db0d1a8ff8ac65.1688165272.git.steadmon@google.com","subject":"[PATCH v5] unit tests: Add a project plan document","fromName":"Josh Steadmon","fromEmail":"steadmon@google.com","sentAt":"2023-08-07T23:07:55Z","receivedAt":"2023-08-07T23:11:25Z","isPatch":true,"sender":{"key":"steadmon@google.com","avatar":"https://avatars.githubusercontent.com/u/2654920?v=4"},"body":"In our current testing environment, we spend a significant amount of\neffort crafting end-to-end tests for error conditions that could easily\nbe captured by unit tests (or we simply forgo some hard-to-setup and\nrare error conditions). Describe what we hope to accomplish by\nimplementing unit tests, and explain some open questions and milestones.\nDiscuss desired features for test frameworks/harnesses, and provide a\npreliminary comparison of several different frameworks.\n\nCoauthored-by: Calvin Wan \nSigned-off-by: Calvin Wan \nSigned-off-by: Josh Steadmon \n---\nIn our current testing environment, we spend a significant amount of\neffort crafting end-to-end tests for error conditions that could easily\nbe captured by unit tests (or we simply forgo some hard-to-setup and\nrare error conditions). Unit tests additionally provide stability to the\ncodebase and can simplify debugging through isolation. Turning parts of\nGit into libraries[1] gives us the ability to run unit tests on the\nlibraries and to write unit tests in C. Writing unit tests in pure C,\nrather than with our current shell/test-tool helper setup, simplifies\ntest setup, simplifies passing data around (no shell-isms required), and\nreduces testing runtime by not spawning a separate process for every\ntest invocation.\n\nThis patch adds a project document describing our goals for adding unit\ntests, as well as a discussion of features needed from prospective test\nframeworks or harnesses. It also includes a WIP comparison of various\nproposed frameworks. Later iterations of this series will probably\ninclude a sample unit test and Makefile integration once we've settled\non a framework. A rendered preview of this doc can be found at [2].\n\n[1] https://lore.kernel.org/git/CAJoAoZ=Cig_kLocxKGax31sU7Xe4==BGzC__Bg2_pr7krNq6MA@mail.gmail.com/\n[2] https://github.com/steadmon/git/blob/unit-tests-dev/Documentation/technical/unit-tests.adoc\n\nReviewers can help this series progress by discussing whether it's\nacceptable to rely on `prove` as a test harness for unit tests. We\nsupport this for the current shell tests suite, but it is not strictly\nrequired.\n\nTODOs remaining:\n- Discuss pre-existing harnesses for the current test suite\n- Figure out how to evaluate frameworks on additional OSes such as *BSD\n and NonStop\n\nChanges in v5:\n- Add comparison point \"License\".\n- Discuss feature priorities\n- Drop frameworks:\n - Incompatible licenses: libtap, cmocka\n - Missing source: MyTAP\n - No TAP support: µnit, cmockery, cmockery2, Unity, minunit, CUnit\n- Drop comparison point \"Coverage reports\": this can generally be\n handled by tools such as `gcov` regardless of the framework used.\n- Drop comparison point \"Inline tests\": there didn't seem to be\n strong interest from reviewers for this feature.\n- Drop comparison point \"Scheduling / re-running\": this was not\n supported by any of the main contenders, and is generally better\n handled by the harness rather than framework.\n- Drop comparison point \"Lazy test planning\": this was supported by\n all frameworks that provide TAP output.\n\nChanges in v4:\n- Add link anchors for the framework comparison dimensions\n- Explain \"Partial\" results for each dimension\n- Use consistent dimension names in the section headers and comparison\n tables\n- Add \"Project KLOC\", \"Adoption\", and \"Inline tests\" dimensions\n- Fill in a few of the missing entries in the comparison table\n\nChanges in v3:\n- Expand the doc with discussion of desired features and a WIP\n comparison.\n- Drop all implementation patches until a framework is selected.\n- Link to v2: https://lore.kernel.org/r/20230517-unit-tests-v2-v2-0-21b5b60f4b32@google.com\n\n Documentation/Makefile | 1 +\n Documentation/technical/unit-tests.txt | 217 +++++++++++++++++++++++++\n 2 files changed, 218 insertions(+)\n create mode 100644 Documentation/technical/unit-tests.txt\n\ndiff --git a/Documentation/Makefile b/Documentation/Makefile\nindex b629176d7d..3f2383a12c 100644\n--- a/Documentation/Makefile\n+++ b/Documentation/Makefile\n@@ -122,6 +122,7 @@ TECH_DOCS += technical/scalar\n TECH_DOCS += technical/send-pack-pipeline\n TECH_DOCS += technical/shallow\n TECH_DOCS += technical/trivial-merge\n+TECH_DOCS += technical/unit-tests\n SP_ARTICLES += $(TECH_DOCS)\n SP_ARTICLES += technical/api-index\n \ndiff --git a/Documentation/technical/unit-tests.txt b/Documentation/technical/unit-tests.txt\nnew file mode 100644\nindex 0000000000..fad9ec9279\n--- /dev/null\n+++ b/Documentation/technical/unit-tests.txt\n@@ -0,0 +1,217 @@\n+= Unit Testing\n+\n+In our current testing environment, we spend a significant amount of effort\n+crafting end-to-end tests for error conditions that could easily be captured by\n+unit tests (or we simply forgo some hard-to-setup and rare error conditions).\n+Unit tests additionally provide stability to the codebase and can simplify\n+debugging through isolation. Writing unit tests in pure C, rather than with our\n+current shell/test-tool helper setup, simplifies test setup, simplifies passing\n+data around (no shell-isms required), and reduces testing runtime by not\n+spawning a separate process for every test invocation.\n+\n+We believe that a large body of unit tests, living alongside the existing test\n+suite, will improve code quality for the Git project.\n+\n+== Definitions\n+\n+For the purposes of this document, we'll use *test framework* to refer to\n+projects that support writing test cases and running tests within the context\n+of a single executable. *Test harness* will refer to projects that manage\n+running multiple executables (each of which may contain multiple test cases) and\n+aggregating their results.\n+\n+In reality, these terms are not strictly defined, and many of the projects\n+discussed below contain features from both categories.\n+\n+For now, we will evaluate projects solely on their framework features. Since we\n+are relying on having TAP output (see below), we can assume that any framework\n+can be made to work with a harness that we can choose later.\n+\n+\n+== Choosing a framework\n+\n+=== Desired features & feature priority\n+\n+There are a variety of features we can use to rank the candidate frameworks, and\n+those features have different priorities:\n+\n+* Critical features: we probably won't consider a framework without these\n+** Can we legally / easily use the project?\n+*** <>\n+*** <>\n+*** <>\n+*** <>\n+** Does the project support our bare-minimum needs?\n+*** <>\n+*** <>\n+*** <>\n+* Nice-to-have features:\n+** <>\n+** <>\n+** <>\n+* Tie-breaker stats\n+** <>\n+** <>\n+\n+[[license]]\n+==== License\n+\n+We must be able to legally use the framework in connection with Git. As Git is\n+licensed only under GPLv2, we must eliminate any LGPLv3, GPLv3, or Apache 2.0\n+projects.\n+\n+[[vendorable-or-ubiquitous]]\n+==== Vendorable or ubiquitous\n+\n+We want to avoid forcing Git developers to install new tools just to run unit\n+tests. Any prospective frameworks and harnesses must either be vendorable\n+(meaning, we can copy their source directly into Git's repository), or so\n+ubiquitous that it is reasonable to expect that most developers will have the\n+tools installed already.\n+\n+[[maintainable-extensible]]\n+==== Maintainable / extensible\n+\n+It is unlikely that any pre-existing project perfectly fits our needs, so any\n+project we select will need to be actively maintained and open to accepting\n+changes. Alternatively, assuming we are vendoring the source into our repo, it\n+must be simple enough that Git developers can feel comfortable making changes as\n+needed to our version.\n+\n+In the comparison table below, \"True\" means that the framework seems to have\n+active developers, that it is simple enough that Git developers can make changes\n+to it, and that the project seems open to accepting external contributions (or\n+that it is vendorable). \"Partial\" means that at least one of the above\n+conditions holds.\n+\n+[[major-platform-support]]\n+==== Major platform support\n+\n+At a bare minimum, unit-testing must work on Linux, MacOS, and Windows.\n+\n+In the comparison table below, \"True\" means that it works on all three major\n+platforms with no issues. \"Partial\" means that there may be annoyances on one or\n+more platforms, but it is still usable in principle.\n+\n+[[tap-support]]\n+==== TAP support\n+\n+The https://testanything.org/[Test Anything Protocol] is a text-based interface\n+that allows tests to communicate with a test harness. It is already used by\n+Git's integration test suite. Supporting TAP output is a mandatory feature for\n+any prospective test framework.\n+\n+In the comparison table below, \"True\" means this is natively supported.\n+\"Partial\" means TAP output must be generated by post-processing the native\n+output.\n+\n+Frameworks that do not have at least Partial support will not be evaluated\n+further.\n+\n+[[diagnostic-output]]\n+==== Diagnostic output\n+\n+When a test case fails, the framework must generate enough diagnostic output to\n+help developers find the appropriate test case in source code in order to debug\n+the failure.\n+\n+[[runtime-skippable-tests]]\n+==== Runtime-skippable tests\n+\n+Test authors may wish to skip certain test cases based on runtime circumstances,\n+so the framework should support this.\n+\n+[[parallel-execution]]\n+==== Parallel execution\n+\n+Ideally, we will build up a significant collection of unit test cases, most\n+likely split across multiple executables. It will be necessary to run these\n+tests in parallel to enable fast develop-test-debug cycles.\n+\n+In the comparison table below, \"True\" means that individual test cases within a\n+single test executable can be run in parallel. We assume that executable-level\n+parallelism can be handled by the test harness.\n+\n+[[mock-support]]\n+==== Mock support\n+\n+Unit test authors may wish to test code that interacts with objects that may be\n+inconvenient to handle in a test (e.g. interacting with a network service).\n+Mocking allows test authors to provide a fake implementation of these objects\n+for more convenient tests.\n+\n+[[signal-error-handling]]\n+==== Signal & error handling\n+\n+The test framework should fail gracefully when test cases are themselves buggy\n+or when they are interrupted by signals during runtime.\n+\n+[[project-kloc]]\n+==== Project KLOC\n+\n+The size of the project, in thousands of lines of code as measured by\n+https://dwheeler.com/sloccount/[sloccount] (rounded up to the next multiple of\n+1,000). As a tie-breaker, we probably prefer a project with fewer LOC.\n+\n+[[adoption]]\n+==== Adoption\n+\n+As a tie-breaker, we prefer a more widely-used project. We use the number of\n+GitHub / GitLab stars to estimate this.\n+\n+\n+=== Comparison\n+\n+[format=\"csv\",options=\"header\",width=\"33%\"]\n+|=====\n+Framework,\"<>\",\"<>\",\"<>\",\"<>\",\"<>\",\"<>\",\"<>\",\"<>\",\"<>\",\"<>\",\"<>\",\"<>\"\n+https://lore.kernel.org/git/c902a166-98ce-afba-93f2-ea6027557176@gmail.com/[Custom Git impl.],[lime-background]#GPL v2#,[lime-background]#True#,[lime-background]#True#,[lime-background]#True#,[lime-background]#True#,[lime-background]#True#,[lime-background]#True#,[red-background]#False#,[red-background]#False#,[red-background]#False#,1,0\n+https://github.com/silentbicycle/greatest[Greatest],[lime-background]#ISC#,[lime-background]#True#,[yellow-background]#Partial#,[lime-background]#True#,[yellow-background]#Partial#,[lime-background]#True#,[lime-background]#True#,[red-background]#False#,[red-background]#False#,[red-background]#False#,3,1400\n+https://github.com/Snaipe/Criterion[Criterion],[lime-background]#MIT#,[red-background]#False#,[yellow-background]#Partial#,[lime-background]#True#,[lime-background]#True#,[lime-background]#True#,[lime-background]#True#,[lime-background]#True#,[red-background]#False#,[lime-background]#True#,19,1800\n+https://github.com/rra/c-tap-harness/[C TAP],[lime-background]#Expat#,[lime-background]#True#,[yellow-background]#Partial#,[yellow-background]#Partial#,[lime-background]#True#,[red-background]#False#,[lime-background]#True#,[red-background]#False#,[red-background]#False#,[red-background]#False#,4,33\n+https://libcheck.github.io/check/[Check],[lime-background]#LGPL v2.1#,[red-background]#False#,[yellow-background]#Partial#,[lime-background]#True#,[lime-background]#True#,[lime-background]#True#,[red-background]#False#,[red-background]#False#,[red-background]#False#,[lime-background]#True#,17,973\n+|=====\n+\n+==== Alternatives considered\n+\n+Several suggested frameworks have been eliminated from consideration:\n+\n+* Incompatible licenses:\n+** https://github.com/zorgnax/libtap[libtap] (LGPL v3)\n+** https://cmocka.org/[cmocka] (Apache 2.0)\n+* Missing source: https://www.kindahl.net/mytap/doc/index.html[MyTap]\n+* No TAP support:\n+** https://nemequ.github.io/munit/[µnit]\n+** https://github.com/google/cmockery[cmockery]\n+** https://github.com/lpabon/cmockery2[cmockery2]\n+** https://github.com/ThrowTheSwitch/Unity[Unity]\n+** https://github.com/siu/minunit[minunit]\n+** https://cunit.sourceforge.net/[CUnit]\n+\n+==== Suggested framework\n+\n+Considering the feature priorities and comparison listed above, a custom\n+framework seems to be the best option.\n+\n+\n+== Choosing a test harness\n+\n+During upstream discussion, it was occasionally noted that `prove` provides many\n+convenient features. While we already support the use of `prove` as a test\n+harness for the shell tests, it is not strictly required.\n+\n+IMPORTANT: It is an open question whether or not we wish to rely on `prove` as a\n+strict dependency for running unit tests.\n+\n+\n+== Milestones\n+\n+* Get upstream agreement on implementing a custom test framework\n+* Determine if it's OK to rely on `prove` for running unit tests\n+* Add useful tests of library-like code\n+* Integrate with Makefile\n+* Integrate with CI\n+* Integrate with\n+ https://lore.kernel.org/git/20230502211454.1673000-1-calvinwan@google.com/[stdlib\n+ work]\n+* Run alongside regular `make test` target\n\nbase-commit: a9e066fa63149291a55f383cfa113d8bdbdaa6b3\n-- \n2.41.0.640.ga95def55d0-goog\n\n"},{"id":"480645","messageId":"90651b2b-6a7a-2ed0-716a-fc87adc957ba@gmail.com","threadId":"59937","inReplyTo":"c7dca1a805a16fd4fd68e86efeec97510e3ac4b8.1691449216.git.steadmon@google.com","subject":"Re: [PATCH v5] unit tests: Add a project plan document","fromName":"Phillip Wood","fromEmail":"phillip.wood123@gmail.com","sentAt":"2023-08-14T13:29:35Z","receivedAt":"2023-08-14T13:30:29Z","isPatch":true,"sender":{"key":"phillip.wood@dunelm.org.uk","avatar":null},"body":"Hi Josh\n\nOn 08/08/2023 00:07, Josh Steadmon wrote:\n> \n> Reviewers can help this series progress by discussing whether it's\n> acceptable to rely on `prove` as a test harness for unit tests. We\n> support this for the current shell tests suite, but it is not strictly\n> required.\n\nIf possible it would be good to be able to run individual test programs \nwithout a harness as we can for our integration tests. For running more \nthan one test program in parallel I think it is fine to require a harness.\n\nI don't have a strong preference for which harness we use so long as it \nprovides a way to (a) run tests that previously failed tests first and \n(b) run slow tests first. I do have a strong preference for using the \nsame harness for both the unit tests and the integration tests so \ndevelopers don't have to learn two different tools. Unless there is a \nproblem with prove it would probably make sense just to keep using that \nas the project test harness.\n\n> TODOs remaining:\n> - Discuss pre-existing harnesses for the current test suite\n> - Figure out how to evaluate frameworks on additional OSes such as *BSD\n> and NonStop\n\nWe have .cirrus.yml in tree which I think gitgitgadget uses to run our \ntest suite on freebsd so we could leverage that for the unit tests. As \nfor NonStop I think we'd just have to rely on Randall running the tests \nas he does now for the integration tests.\n\n> Changes in v5:\n> - Add comparison point \"License\".\n> - Discuss feature priorities\n> - Drop frameworks:\n> - Incompatible licenses: libtap, cmocka\n> - Missing source: MyTAP\n> - No TAP support: µnit, cmockery, cmockery2, Unity, minunit, CUnit\n> - Drop comparison point \"Coverage reports\": this can generally be\n> handled by tools such as `gcov` regardless of the framework used.\n> - Drop comparison point \"Inline tests\": there didn't seem to be\n> strong interest from reviewers for this feature.\n> - Drop comparison point \"Scheduling / re-running\": this was not\n> supported by any of the main contenders, and is generally better\n> handled by the harness rather than framework.\n> - Drop comparison point \"Lazy test planning\": this was supported by\n> all frameworks that provide TAP output.\n\nThese changes all sound sensible to me\n\nThe trimmed down table is most welcome. The custom implementation \nsupports partial parallel execution. It shouldn't be too difficult to \nadd some signal handling to it depending on what we want it to do.\n\nIt sounds like we're getting to the point where we have pinned down our \nrequirements and the available alternatives well enough to make a decision.\n\nBest Wishes\n\nPhillip\n\n"},{"id":"480692","messageId":"ZNwCSdrbuG3K/iMP@google.com","threadId":"59937","inReplyTo":"90651b2b-6a7a-2ed0-716a-fc87adc957ba@gmail.com","subject":"Re: [PATCH v5] unit tests: Add a project plan document","fromName":"Josh Steadmon","fromEmail":"steadmon@google.com","sentAt":"2023-08-15T22:55:05Z","receivedAt":"2023-08-15T22:55:48Z","isPatch":true,"sender":{"key":"steadmon@google.com","avatar":"https://avatars.githubusercontent.com/u/2654920?v=4"},"body":"On 2023.08.14 14:29, Phillip Wood wrote:\n> Hi Josh\n> \n> On 08/08/2023 00:07, Josh Steadmon wrote:\n> > \n> > Reviewers can help this series progress by discussing whether it's\n> > acceptable to rely on `prove` as a test harness for unit tests. We\n> > support this for the current shell tests suite, but it is not strictly\n> > required.\n> \n> If possible it would be good to be able to run individual test programs\n> without a harness as we can for our integration tests. For running more than\n> one test program in parallel I think it is fine to require a harness.\n\nSounds good. This is working in v6 which I hope to send to the list\nsoon.\n\n\n> I don't have a strong preference for which harness we use so long as it\n> provides a way to (a) run tests that previously failed tests first and (b)\n> run slow tests first. I do have a strong preference for using the same\n> harness for both the unit tests and the integration tests so developers\n> don't have to learn two different tools. Unless there is a problem with\n> prove it would probably make sense just to keep using that as the project\n> test harness.\n\nTo be clear, it sounds like both of these can be done with `prove`\n(using the various --state settings) without any further support from\nour unit tests, right? I see that we do have a \"failed\" target for\nre-running integration tests, but that relies on some test-lib.sh\nfeatures that currently have no equivalent in the unit test framework.\n\n\n> > TODOs remaining:\n> > - Discuss pre-existing harnesses for the current test suite\n> > - Figure out how to evaluate frameworks on additional OSes such as *BSD\n> > and NonStop\n> \n> We have .cirrus.yml in tree which I think gitgitgadget uses to run our test\n> suite on freebsd so we could leverage that for the unit tests. As for\n> NonStop I think we'd just have to rely on Randall running the tests as he\n> does now for the integration tests.\n\nThanks for the pointer. I've updated .cirrus.yml (as well as the GitHub\nCI configs) for v6.\n\n\n> > Changes in v5:\n> > - Add comparison point \"License\".\n> > - Discuss feature priorities\n> > - Drop frameworks:\n> > - Incompatible licenses: libtap, cmocka\n> > - Missing source: MyTAP\n> > - No TAP support: µnit, cmockery, cmockery2, Unity, minunit, CUnit\n> > - Drop comparison point \"Coverage reports\": this can generally be\n> > handled by tools such as `gcov` regardless of the framework used.\n> > - Drop comparison point \"Inline tests\": there didn't seem to be\n> > strong interest from reviewers for this feature.\n> > - Drop comparison point \"Scheduling / re-running\": this was not\n> > supported by any of the main contenders, and is generally better\n> > handled by the harness rather than framework.\n> > - Drop comparison point \"Lazy test planning\": this was supported by\n> > all frameworks that provide TAP output.\n> \n> These changes all sound sensible to me\n> \n> The trimmed down table is most welcome. The custom implementation supports\n> partial parallel execution. It shouldn't be too difficult to add some signal\n> handling to it depending on what we want it to do.\n> \n> It sounds like we're getting to the point where we have pinned down our\n> requirements and the available alternatives well enough to make a decision.\n\nYes, v6 will include your TAP implementation (I assume you are still OK\nif I include your patch in this series?).\n\n> Best Wishes\n> \n> Phillip\n> \n"},{"id":"480730","messageId":"ea33518d00bed2cb3015ba37fe09b507eb986c2e.1692229626.git.steadmon@google.com","threadId":"59937","inReplyTo":"cover.1692229626.git.steadmon@google.com","subject":"[PATCH v6 3/3] ci: run unit tests in CI","fromName":"Josh Steadmon","fromEmail":"steadmon@google.com","sentAt":"2023-08-16T23:50:07Z","receivedAt":"2023-08-16T23:51:15Z","isPatch":true,"sender":{"key":"steadmon@google.com","avatar":"https://avatars.githubusercontent.com/u/2654920?v=4"},"body":"Run unit tests in both Cirrus and GitHub CI. For sharded CI instances\n(currently just Windows on GitHub), run only on the first shard. This is\nOK while we have only a single unit test executable, but we may wish to\ndistribute tests more evenly when we add new unit tests in the future.\n\nWe may also want to add more status output in our unit test framework,\nso that we can do similar post-processing as in\nci/lib.sh:handle_failed_tests().\n\nSigned-off-by: Josh Steadmon \n---\n .cirrus.yml | 2 +-\n ci/run-build-and-tests.sh | 2 ++\n ci/run-test-slice.sh | 5 +++++\n t/Makefile | 2 +-\n 4 files changed, 9 insertions(+), 2 deletions(-)\n\ndiff --git a/.cirrus.yml b/.cirrus.yml\nindex 4860bebd32..b6280692d2 100644\n--- a/.cirrus.yml\n+++ b/.cirrus.yml\n@@ -19,4 +19,4 @@ freebsd_12_task:\n build_script:\n - su git -c gmake\n test_script:\n- - su git -c 'gmake test'\n+ - su git -c 'gmake DEFAULT_UNIT_TEST_TARGET=unit-tests-prove test unit-tests'\ndiff --git a/ci/run-build-and-tests.sh b/ci/run-build-and-tests.sh\nindex 2528f25e31..7a1466b868 100755\n--- a/ci/run-build-and-tests.sh\n+++ b/ci/run-build-and-tests.sh\n@@ -50,6 +50,8 @@ if test -n \"$run_tests\"\n then\n \tgroup \"Run tests\" make test ||\n \thandle_failed_tests\n+\tgroup \"Run unit tests\" \\\n+\t\tmake DEFAULT_UNIT_TEST_TARGET=unit-tests-prove unit-tests\n fi\n check_unignored_build_artifacts\n \ndiff --git a/ci/run-test-slice.sh b/ci/run-test-slice.sh\nindex a3c67956a8..ae8094382f 100755\n--- a/ci/run-test-slice.sh\n+++ b/ci/run-test-slice.sh\n@@ -15,4 +15,9 @@ group \"Run tests\" make --quiet -C t T=\"$(cd t &&\n \ttr '\\n' ' ')\" ||\n handle_failed_tests\n \n+# We only have one unit test at the moment, so run it in the first slice\n+if [ \"$1\" == \"0\" ] ; then\n+\tgroup \"Run unit tests\" make --quiet -C t unit-tests-prove\n+fi\n+\n check_unignored_build_artifacts\ndiff --git a/t/Makefile b/t/Makefile\nindex 2db8b3adb1..095334bfde 100644\n--- a/t/Makefile\n+++ b/t/Makefile\n@@ -42,7 +42,7 @@ TPERF = $(sort $(wildcard perf/p[0-9][0-9][0-9][0-9]-*.sh))\n TINTEROP = $(sort $(wildcard interop/i[0-9][0-9][0-9][0-9]-*.sh))\n CHAINLINTTESTS = $(sort $(patsubst chainlint/%.test,%,$(wildcard chainlint/*.test)))\n CHAINLINT = '$(PERL_PATH_SQ)' chainlint.pl\n-UNIT_TESTS = $(sort $(filter-out %.h %.c %.o unit-tests/t-basic%,$(wildcard unit-tests/*)))\n+UNIT_TESTS = $(sort $(filter-out %.h %.c %.o unit-tests/t-basic%,$(wildcard unit-tests/t-*)))\n \n # `test-chainlint` (which is a dependency of `test-lint`, `test` and `prove`)\n # checks all tests in all scripts via a single invocation, so tell individual\n-- \n2.41.0.694.ge786442a9b-goog\n\n"},{"id":"480731","messageId":"cover.1692229626.git.steadmon@google.com","threadId":"59937","inReplyTo":"0169ce6fb9ccafc089b74ae406db0d1a8ff8ac65.1688165272.git.steadmon@google.com","subject":"[PATCH v6 0/3] Add unit test framework and project plan MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit","fromName":"Josh Steadmon","fromEmail":"steadmon@google.com","sentAt":"2023-08-16T23:50:04Z","receivedAt":"2023-08-16T23:51:15Z","isPatch":true,"sender":{"key":"steadmon@google.com","avatar":"https://avatars.githubusercontent.com/u/2654920?v=4"},"body":"In our current testing environment, we spend a significant amount of\neffort crafting end-to-end tests for error conditions that could easily\nbe captured by unit tests (or we simply forgo some hard-to-setup and\nrare error conditions). Unit tests additionally provide stability to the\ncodebase and can simplify debugging through isolation. Turning parts of\nGit into libraries[1] gives us the ability to run unit tests on the\nlibraries and to write unit tests in C. Writing unit tests in pure C,\nrather than with our current shell/test-tool helper setup, simplifies\ntest setup, simplifies passing data around (no shell-isms required), and\nreduces testing runtime by not spawning a separate process for every\ntest invocation.\n\nThis series begins with a project document covering our goals for adding\nunit tests and a discussion of alternative frameworks considered, as\nwell as the features used to evaluate them. A rendered preview of this\ndoc can be found at [2]. It also adds Phillip Wood's TAP implemenation\n(with some slightly re-worked Makefile rules) and a sample strbuf unit\ntest. Finally, we modify the configs for GitHub and Cirrus CI to run the\nunit tests. Sample runs showing successful CI runs can be found at [3],\n[4], and [5].\n\n[1] https://lore.kernel.org/git/CAJoAoZ=Cig_kLocxKGax31sU7Xe4==BGzC__Bg2_pr7krNq6MA@mail.gmail.com/\n[2] https://github.com/steadmon/git/blob/unit-tests-asciidoc/Documentation/technical/unit-tests.adoc\n[3] https://github.com/steadmon/git/actions/runs/5884659246/job/15959781385#step:4:1803\n[4] https://github.com/steadmon/git/actions/runs/5884659246/job/15959938401#step:5:186\n[5] https://cirrus-ci.com/task/6126304366428160 (unrelated tests failed,\n but note that t-strbuf ran successfully)\n\nIn addition to reviewing the patches in this series, reviewers can help\nthis series progress by chiming in on these remaining TODOs:\n- Figure out how to ensure tests run on additional OSes such as NonStop\n- Figure out if we should collect unit tests statistics similar to the\n \"counts\" files for shell tests\n- Decide if it's OK to wait on sharding unit tests across \"sliced\" CI\n instances\n- Provide guidelines for writing new unit tests\n\nChanges in v6:\n- Officially recommend using Phillip Wood's TAP framework\n- Add an example strbuf unit test using the TAP framework as well as\n Makefile integration\n- Run unit tests in CI\n\nChanges in v5:\n- Add comparison point \"License\".\n- Discuss feature priorities\n- Drop frameworks:\n - Incompatible licenses: libtap, cmocka\n - Missing source: MyTAP\n - No TAP support: µnit, cmockery, cmockery2, Unity, minunit, CUnit\n- Drop comparison point \"Coverage reports\": this can generally be\n handled by tools such as `gcov` regardless of the framework used.\n- Drop comparison point \"Inline tests\": there didn't seem to be\n strong interest from reviewers for this feature.\n- Drop comparison point \"Scheduling / re-running\": this was not\n supported by any of the main contenders, and is generally better\n handled by the harness rather than framework.\n- Drop comparison point \"Lazy test planning\": this was supported by\n all frameworks that provide TAP output.\n\nChanges in v4:\n- Add link anchors for the framework comparison dimensions\n- Explain \"Partial\" results for each dimension\n- Use consistent dimension names in the section headers and comparison\n tables\n- Add \"Project KLOC\", \"Adoption\", and \"Inline tests\" dimensions\n- Fill in a few of the missing entries in the comparison table\n\nChanges in v3:\n- Expand the doc with discussion of desired features and a WIP\n comparison.\n- Drop all implementation patches until a framework is selected.\n- Link to v2: https://lore.kernel.org/r/20230517-unit-tests-v2-v2-0-21b5b60f4b32@google.com\n\n\nJosh Steadmon (2):\n unit tests: Add a project plan document\n ci: run unit tests in CI\n\nPhillip Wood (1):\n unit tests: add TAP unit test framework\n\n .cirrus.yml | 2 +-\n Documentation/Makefile | 1 +\n Documentation/technical/unit-tests.txt | 220 +++++++++++++++++\n Makefile | 24 +-\n ci/run-build-and-tests.sh | 2 +\n ci/run-test-slice.sh | 5 +\n t/Makefile | 15 +-\n t/t0080-unit-test-output.sh | 58 +++++\n t/unit-tests/.gitignore | 2 +\n t/unit-tests/t-basic.c | 95 +++++++\n t/unit-tests/t-strbuf.c | 75 ++++++\n t/unit-tests/test-lib.c | 329 +++++++++++++++++++++++++\n t/unit-tests/test-lib.h | 143 +++++++++++\n 13 files changed, 966 insertions(+), 5 deletions(-)\n create mode 100644 Documentation/technical/unit-tests.txt\n create mode 100755 t/t0080-unit-test-output.sh\n create mode 100644 t/unit-tests/.gitignore\n create mode 100644 t/unit-tests/t-basic.c\n create mode 100644 t/unit-tests/t-strbuf.c\n create mode 100644 t/unit-tests/test-lib.c\n create mode 100644 t/unit-tests/test-lib.h\n\nRange-diff against v5:\n1: c7dca1a805 ! 1: 81c5148a12 unit tests: Add a project plan document\n @@ Commit message\n preliminary comparison of several different frameworks.\n \n - Coauthored-by: Calvin Wan \n + Co-authored-by: Calvin Wan \n Signed-off-by: Calvin Wan \n Signed-off-by: Josh Steadmon \n \n @@ Documentation/technical/unit-tests.txt (new)\n +\n +== Choosing a framework\n +\n -+=== Desired features & feature priority\n ++We believe the best option is to implement a custom TAP framework for the Git\n ++project. We use a version of the framework originally proposed in\n ++https://lore.kernel.org/git/c902a166-98ce-afba-93f2-ea6027557176@gmail.com/[1].\n ++\n ++\n ++== Choosing a test harness\n ++\n ++During upstream discussion, it was occasionally noted that `prove` provides many\n ++convenient features, such as scheduling slower tests first, or re-running\n ++previously failed tests.\n ++\n ++While we already support the use of `prove` as a test harness for the shell\n ++tests, it is not strictly required. The t/Makefile allows running shell tests\n ++directly (though with interleaved output if parallelism is enabled). Git\n ++developers who wish to use `prove` as a more advanced harness can do so by\n ++setting DEFAULT_TEST_TARGET=prove in their config.mak.\n ++\n ++We will follow a similar approach for unit tests: by default the test\n ++executables will be run directly from the t/Makefile, but `prove` can be\n ++configured with DEFAULT_UNIT_TEST_TARGET=prove.\n ++\n ++\n ++== Framework selection\n +\n +There are a variety of features we can use to rank the candidate frameworks, and\n +those features have different priorities:\n @@ Documentation/technical/unit-tests.txt (new)\n +** <>\n +\n +[[license]]\n -+==== License\n ++=== License\n +\n +We must be able to legally use the framework in connection with Git. As Git is\n +licensed only under GPLv2, we must eliminate any LGPLv3, GPLv3, or Apache 2.0\n +projects.\n +\n +[[vendorable-or-ubiquitous]]\n -+==== Vendorable or ubiquitous\n ++=== Vendorable or ubiquitous\n +\n +We want to avoid forcing Git developers to install new tools just to run unit\n +tests. Any prospective frameworks and harnesses must either be vendorable\n @@ Documentation/technical/unit-tests.txt (new)\n +tools installed already.\n +\n +[[maintainable-extensible]]\n -+==== Maintainable / extensible\n ++=== Maintainable / extensible\n +\n +It is unlikely that any pre-existing project perfectly fits our needs, so any\n +project we select will need to be actively maintained and open to accepting\n @@ Documentation/technical/unit-tests.txt (new)\n +conditions holds.\n +\n +[[major-platform-support]]\n -+==== Major platform support\n ++=== Major platform support\n +\n +At a bare minimum, unit-testing must work on Linux, MacOS, and Windows.\n +\n @@ Documentation/technical/unit-tests.txt (new)\n +more platforms, but it is still usable in principle.\n +\n +[[tap-support]]\n -+==== TAP support\n ++=== TAP support\n +\n +The https://testanything.org/[Test Anything Protocol] is a text-based interface\n +that allows tests to communicate with a test harness. It is already used by\n @@ Documentation/technical/unit-tests.txt (new)\n +further.\n +\n +[[diagnostic-output]]\n -+==== Diagnostic output\n ++=== Diagnostic output\n +\n +When a test case fails, the framework must generate enough diagnostic output to\n +help developers find the appropriate test case in source code in order to debug\n +the failure.\n +\n +[[runtime-skippable-tests]]\n -+==== Runtime-skippable tests\n ++=== Runtime-skippable tests\n +\n +Test authors may wish to skip certain test cases based on runtime circumstances,\n +so the framework should support this.\n +\n +[[parallel-execution]]\n -+==== Parallel execution\n ++=== Parallel execution\n +\n +Ideally, we will build up a significant collection of unit test cases, most\n +likely split across multiple executables. It will be necessary to run these\n @@ Documentation/technical/unit-tests.txt (new)\n +parallelism can be handled by the test harness.\n +\n +[[mock-support]]\n -+==== Mock support\n ++=== Mock support\n +\n +Unit test authors may wish to test code that interacts with objects that may be\n +inconvenient to handle in a test (e.g. interacting with a network service).\n @@ Documentation/technical/unit-tests.txt (new)\n +for more convenient tests.\n +\n +[[signal-error-handling]]\n -+==== Signal & error handling\n ++=== Signal & error handling\n +\n +The test framework should fail gracefully when test cases are themselves buggy\n +or when they are interrupted by signals during runtime.\n +\n +[[project-kloc]]\n -+==== Project KLOC\n ++=== Project KLOC\n +\n +The size of the project, in thousands of lines of code as measured by\n +https://dwheeler.com/sloccount/[sloccount] (rounded up to the next multiple of\n +1,000). As a tie-breaker, we probably prefer a project with fewer LOC.\n +\n +[[adoption]]\n -+==== Adoption\n ++=== Adoption\n +\n +As a tie-breaker, we prefer a more widely-used project. We use the number of\n +GitHub / GitLab stars to estimate this.\n @@ Documentation/technical/unit-tests.txt (new)\n +https://libcheck.github.io/check/[Check],[lime-background]#LGPL v2.1#,[red-background]#False#,[yellow-background]#Partial#,[lime-background]#True#,[lime-background]#True#,[lime-background]#True#,[red-background]#False#,[red-background]#False#,[red-background]#False#,[lime-background]#True#,17,973\n +|=====\n +\n -+==== Alternatives considered\n ++=== Additional framework candidates\n +\n +Several suggested frameworks have been eliminated from consideration:\n +\n @@ Documentation/technical/unit-tests.txt (new)\n +** https://github.com/siu/minunit[minunit]\n +** https://cunit.sourceforge.net/[CUnit]\n +\n -+==== Suggested framework\n -+\n -+Considering the feature priorities and comparison listed above, a custom\n -+framework seems to be the best option.\n -+\n -+\n -+== Choosing a test harness\n -+\n -+During upstream discussion, it was occasionally noted that `prove` provides many\n -+convenient features. While we already support the use of `prove` as a test\n -+harness for the shell tests, it is not strictly required.\n -+\n -+IMPORTANT: It is an open question whether or not we wish to rely on `prove` as a\n -+strict dependency for running unit tests.\n -+\n +\n +== Milestones\n +\n -+* Get upstream agreement on implementing a custom test framework\n -+* Determine if it's OK to rely on `prove` for running unit tests\n +* Add useful tests of library-like code\n -+* Integrate with Makefile\n -+* Integrate with CI\n +* Integrate with\n + https://lore.kernel.org/git/20230502211454.1673000-1-calvinwan@google.com/[stdlib\n + work]\n-: ---------- > 2: ca284c575e unit tests: add TAP unit test framework\n-: ---------- > 3: ea33518d00 ci: run unit tests in CI\n\nbase-commit: a9e066fa63149291a55f383cfa113d8bdbdaa6b3\n-- \n2.41.0.694.ge786442a9b-goog\n\n"},{"id":"480732","messageId":"81c5148a1267b8f9ce432a950340f0fa16b4d773.1692229626.git.steadmon@google.com","threadId":"59937","inReplyTo":"cover.1692229626.git.steadmon@google.com","subject":"[PATCH v6 1/3] unit tests: Add a project plan document","fromName":"Josh Steadmon","fromEmail":"steadmon@google.com","sentAt":"2023-08-16T23:50:05Z","receivedAt":"2023-08-16T23:51:15Z","isPatch":true,"sender":{"key":"steadmon@google.com","avatar":"https://avatars.githubusercontent.com/u/2654920?v=4"},"body":"In our current testing environment, we spend a significant amount of\neffort crafting end-to-end tests for error conditions that could easily\nbe captured by unit tests (or we simply forgo some hard-to-setup and\nrare error conditions). Describe what we hope to accomplish by\nimplementing unit tests, and explain some open questions and milestones.\nDiscuss desired features for test frameworks/harnesses, and provide a\npreliminary comparison of several different frameworks.\n\nCo-authored-by: Calvin Wan \nSigned-off-by: Calvin Wan \nSigned-off-by: Josh Steadmon \n---\n Documentation/Makefile | 1 +\n Documentation/technical/unit-tests.txt | 220 +++++++++++++++++++++++++\n 2 files changed, 221 insertions(+)\n create mode 100644 Documentation/technical/unit-tests.txt\n\ndiff --git a/Documentation/Makefile b/Documentation/Makefile\nindex b629176d7d..3f2383a12c 100644\n--- a/Documentation/Makefile\n+++ b/Documentation/Makefile\n@@ -122,6 +122,7 @@ TECH_DOCS += technical/scalar\n TECH_DOCS += technical/send-pack-pipeline\n TECH_DOCS += technical/shallow\n TECH_DOCS += technical/trivial-merge\n+TECH_DOCS += technical/unit-tests\n SP_ARTICLES += $(TECH_DOCS)\n SP_ARTICLES += technical/api-index\n \ndiff --git a/Documentation/technical/unit-tests.txt b/Documentation/technical/unit-tests.txt\nnew file mode 100644\nindex 0000000000..b7a89cc838\n--- /dev/null\n+++ b/Documentation/technical/unit-tests.txt\n@@ -0,0 +1,220 @@\n+= Unit Testing\n+\n+In our current testing environment, we spend a significant amount of effort\n+crafting end-to-end tests for error conditions that could easily be captured by\n+unit tests (or we simply forgo some hard-to-setup and rare error conditions).\n+Unit tests additionally provide stability to the codebase and can simplify\n+debugging through isolation. Writing unit tests in pure C, rather than with our\n+current shell/test-tool helper setup, simplifies test setup, simplifies passing\n+data around (no shell-isms required), and reduces testing runtime by not\n+spawning a separate process for every test invocation.\n+\n+We believe that a large body of unit tests, living alongside the existing test\n+suite, will improve code quality for the Git project.\n+\n+== Definitions\n+\n+For the purposes of this document, we'll use *test framework* to refer to\n+projects that support writing test cases and running tests within the context\n+of a single executable. *Test harness* will refer to projects that manage\n+running multiple executables (each of which may contain multiple test cases) and\n+aggregating their results.\n+\n+In reality, these terms are not strictly defined, and many of the projects\n+discussed below contain features from both categories.\n+\n+For now, we will evaluate projects solely on their framework features. Since we\n+are relying on having TAP output (see below), we can assume that any framework\n+can be made to work with a harness that we can choose later.\n+\n+\n+== Choosing a framework\n+\n+We believe the best option is to implement a custom TAP framework for the Git\n+project. We use a version of the framework originally proposed in\n+https://lore.kernel.org/git/c902a166-98ce-afba-93f2-ea6027557176@gmail.com/[1].\n+\n+\n+== Choosing a test harness\n+\n+During upstream discussion, it was occasionally noted that `prove` provides many\n+convenient features, such as scheduling slower tests first, or re-running\n+previously failed tests.\n+\n+While we already support the use of `prove` as a test harness for the shell\n+tests, it is not strictly required. The t/Makefile allows running shell tests\n+directly (though with interleaved output if parallelism is enabled). Git\n+developers who wish to use `prove` as a more advanced harness can do so by\n+setting DEFAULT_TEST_TARGET=prove in their config.mak.\n+\n+We will follow a similar approach for unit tests: by default the test\n+executables will be run directly from the t/Makefile, but `prove` can be\n+configured with DEFAULT_UNIT_TEST_TARGET=prove.\n+\n+\n+== Framework selection\n+\n+There are a variety of features we can use to rank the candidate frameworks, and\n+those features have different priorities:\n+\n+* Critical features: we probably won't consider a framework without these\n+** Can we legally / easily use the project?\n+*** <>\n+*** <>\n+*** <>\n+*** <>\n+** Does the project support our bare-minimum needs?\n+*** <>\n+*** <>\n+*** <>\n+* Nice-to-have features:\n+** <>\n+** <>\n+** <>\n+* Tie-breaker stats\n+** <>\n+** <>\n+\n+[[license]]\n+=== License\n+\n+We must be able to legally use the framework in connection with Git. As Git is\n+licensed only under GPLv2, we must eliminate any LGPLv3, GPLv3, or Apache 2.0\n+projects.\n+\n+[[vendorable-or-ubiquitous]]\n+=== Vendorable or ubiquitous\n+\n+We want to avoid forcing Git developers to install new tools just to run unit\n+tests. Any prospective frameworks and harnesses must either be vendorable\n+(meaning, we can copy their source directly into Git's repository), or so\n+ubiquitous that it is reasonable to expect that most developers will have the\n+tools installed already.\n+\n+[[maintainable-extensible]]\n+=== Maintainable / extensible\n+\n+It is unlikely that any pre-existing project perfectly fits our needs, so any\n+project we select will need to be actively maintained and open to accepting\n+changes. Alternatively, assuming we are vendoring the source into our repo, it\n+must be simple enough that Git developers can feel comfortable making changes as\n+needed to our version.\n+\n+In the comparison table below, \"True\" means that the framework seems to have\n+active developers, that it is simple enough that Git developers can make changes\n+to it, and that the project seems open to accepting external contributions (or\n+that it is vendorable). \"Partial\" means that at least one of the above\n+conditions holds.\n+\n+[[major-platform-support]]\n+=== Major platform support\n+\n+At a bare minimum, unit-testing must work on Linux, MacOS, and Windows.\n+\n+In the comparison table below, \"True\" means that it works on all three major\n+platforms with no issues. \"Partial\" means that there may be annoyances on one or\n+more platforms, but it is still usable in principle.\n+\n+[[tap-support]]\n+=== TAP support\n+\n+The https://testanything.org/[Test Anything Protocol] is a text-based interface\n+that allows tests to communicate with a test harness. It is already used by\n+Git's integration test suite. Supporting TAP output is a mandatory feature for\n+any prospective test framework.\n+\n+In the comparison table below, \"True\" means this is natively supported.\n+\"Partial\" means TAP output must be generated by post-processing the native\n+output.\n+\n+Frameworks that do not have at least Partial support will not be evaluated\n+further.\n+\n+[[diagnostic-output]]\n+=== Diagnostic output\n+\n+When a test case fails, the framework must generate enough diagnostic output to\n+help developers find the appropriate test case in source code in order to debug\n+the failure.\n+\n+[[runtime-skippable-tests]]\n+=== Runtime-skippable tests\n+\n+Test authors may wish to skip certain test cases based on runtime circumstances,\n+so the framework should support this.\n+\n+[[parallel-execution]]\n+=== Parallel execution\n+\n+Ideally, we will build up a significant collection of unit test cases, most\n+likely split across multiple executables. It will be necessary to run these\n+tests in parallel to enable fast develop-test-debug cycles.\n+\n+In the comparison table below, \"True\" means that individual test cases within a\n+single test executable can be run in parallel. We assume that executable-level\n+parallelism can be handled by the test harness.\n+\n+[[mock-support]]\n+=== Mock support\n+\n+Unit test authors may wish to test code that interacts with objects that may be\n+inconvenient to handle in a test (e.g. interacting with a network service).\n+Mocking allows test authors to provide a fake implementation of these objects\n+for more convenient tests.\n+\n+[[signal-error-handling]]\n+=== Signal & error handling\n+\n+The test framework should fail gracefully when test cases are themselves buggy\n+or when they are interrupted by signals during runtime.\n+\n+[[project-kloc]]\n+=== Project KLOC\n+\n+The size of the project, in thousands of lines of code as measured by\n+https://dwheeler.com/sloccount/[sloccount] (rounded up to the next multiple of\n+1,000). As a tie-breaker, we probably prefer a project with fewer LOC.\n+\n+[[adoption]]\n+=== Adoption\n+\n+As a tie-breaker, we prefer a more widely-used project. We use the number of\n+GitHub / GitLab stars to estimate this.\n+\n+\n+=== Comparison\n+\n+[format=\"csv\",options=\"header\",width=\"33%\"]\n+|=====\n+Framework,\"<>\",\"<>\",\"<>\",\"<>\",\"<>\",\"<>\",\"<>\",\"<>\",\"<>\",\"<>\",\"<>\",\"<>\"\n+https://lore.kernel.org/git/c902a166-98ce-afba-93f2-ea6027557176@gmail.com/[Custom Git impl.],[lime-background]#GPL v2#,[lime-background]#True#,[lime-background]#True#,[lime-background]#True#,[lime-background]#True#,[lime-background]#True#,[lime-background]#True#,[red-background]#False#,[red-background]#False#,[red-background]#False#,1,0\n+https://github.com/silentbicycle/greatest[Greatest],[lime-background]#ISC#,[lime-background]#True#,[yellow-background]#Partial#,[lime-background]#True#,[yellow-background]#Partial#,[lime-background]#True#,[lime-background]#True#,[red-background]#False#,[red-background]#False#,[red-background]#False#,3,1400\n+https://github.com/Snaipe/Criterion[Criterion],[lime-background]#MIT#,[red-background]#False#,[yellow-background]#Partial#,[lime-background]#True#,[lime-background]#True#,[lime-background]#True#,[lime-background]#True#,[lime-background]#True#,[red-background]#False#,[lime-background]#True#,19,1800\n+https://github.com/rra/c-tap-harness/[C TAP],[lime-background]#Expat#,[lime-background]#True#,[yellow-background]#Partial#,[yellow-background]#Partial#,[lime-background]#True#,[red-background]#False#,[lime-background]#True#,[red-background]#False#,[red-background]#False#,[red-background]#False#,4,33\n+https://libcheck.github.io/check/[Check],[lime-background]#LGPL v2.1#,[red-background]#False#,[yellow-background]#Partial#,[lime-background]#True#,[lime-background]#True#,[lime-background]#True#,[red-background]#False#,[red-background]#False#,[red-background]#False#,[lime-background]#True#,17,973\n+|=====\n+\n+=== Additional framework candidates\n+\n+Several suggested frameworks have been eliminated from consideration:\n+\n+* Incompatible licenses:\n+** https://github.com/zorgnax/libtap[libtap] (LGPL v3)\n+** https://cmocka.org/[cmocka] (Apache 2.0)\n+* Missing source: https://www.kindahl.net/mytap/doc/index.html[MyTap]\n+* No TAP support:\n+** https://nemequ.github.io/munit/[µnit]\n+** https://github.com/google/cmockery[cmockery]\n+** https://github.com/lpabon/cmockery2[cmockery2]\n+** https://github.com/ThrowTheSwitch/Unity[Unity]\n+** https://github.com/siu/minunit[minunit]\n+** https://cunit.sourceforge.net/[CUnit]\n+\n+\n+== Milestones\n+\n+* Add useful tests of library-like code\n+* Integrate with\n+ https://lore.kernel.org/git/20230502211454.1673000-1-calvinwan@google.com/[stdlib\n+ work]\n+* Run alongside regular `make test` target\n-- \n2.41.0.694.ge786442a9b-goog\n\n"},{"id":"480733","messageId":"ca284c575ece0aee7149641d5fb1977ccd7e7873.1692229626.git.steadmon@google.com","threadId":"59937","inReplyTo":"cover.1692229626.git.steadmon@google.com","subject":"[PATCH v6 2/3] unit tests: add TAP unit test framework","fromName":"Josh Steadmon","fromEmail":"steadmon@google.com","sentAt":"2023-08-16T23:50:06Z","receivedAt":"2023-08-16T23:51:15Z","isPatch":true,"sender":{"key":"steadmon@google.com","avatar":"https://avatars.githubusercontent.com/u/2654920?v=4"},"body":"From: Phillip Wood \n\nThis patch contains an implementation for writing unit tests with TAP\noutput. Each test is a function that contains one or more checks. The\ntest is run with the TEST() macro and if any of the checks fail then the\ntest will fail. A complete program that tests STRBUF_INIT would look\nlike\n\n #include \"test-lib.h\"\n #include \"strbuf.h\"\n\n static void t_static_init(void)\n {\n struct strbuf buf = STRBUF_INIT;\n\n check_uint(buf.len, ==, 0);\n check_uint(buf.alloc, ==, 0);\n if (check(buf.buf == strbuf_slopbuf))\n\t\t return; /* avoid SIGSEV */\n check_char(buf.buf[0], ==, '\\0');\n }\n\n int main(void)\n {\n TEST(t_static_init(), \"static initialization works);\n\n return test_done();\n }\n\nThe output of this program would be\n\n ok 1 - static initialization works\n 1..1\n\nIf any of the checks in a test fail then they print a diagnostic message\nto aid debugging and the test will be reported as failing. For example a\nfailing integer check would look like\n\n # check \"x >= 3\" failed at my-test.c:102\n # left: 2\n # right: 3\n not ok 1 - x is greater than or equal to three\n\nThere are a number of check functions implemented so far. check() checks\na boolean condition, check_int(), check_uint() and check_char() take two\nvalues to compare and a comparison operator. check_str() will check if\ntwo strings are equal. Custom checks are simple to implement as shown in\nthe comments above test_assert() in test-lib.h.\n\nTests can be skipped with test_skip() which can be supplied with a\nreason for skipping which it will print. Tests can print diagnostic\nmessages with test_msg(). Checks that are known to fail can be wrapped\nin TEST_TODO().\n\nThere are a couple of example test programs included in this\npatch. t-basic.c implements some self-tests and demonstrates the\ndiagnostic output for failing test. The output of this program is\nchecked by t0080-unit-test-output.sh. t-strbuf.c shows some example\nunit tests for strbuf.c\n\nThe unit tests can be built with \"make unit-tests\" (this works but the\nMakefile changes need some further work). Once they have been built they\ncan be run manually (e.g t/unit-tests/t-strbuf) or with prove.\n\nSigned-off-by: Phillip Wood \nSigned-off-by: Josh Steadmon \n\ndiff --git a/Makefile b/Makefile\nindex e440728c24..4016da6e39 100644\n\n--- a/Makefile\n+++ b/Makefile\n@@ -682,6 +682,8 @@ TEST_BUILTINS_OBJS =\n TEST_OBJS =\n TEST_PROGRAMS_NEED_X =\n THIRD_PARTY_SOURCES =\n+UNIT_TEST_PROGRAMS =\n+UNIT_TEST_DIR = t/unit-tests\n\n # Having this variable in your environment would break pipelines because\n # you cause \"cd\" to echo its destination to stdout. It can also take\n@@ -1331,6 +1333,12 @@ THIRD_PARTY_SOURCES += compat/regex/%\n THIRD_PARTY_SOURCES += sha1collisiondetection/%\n THIRD_PARTY_SOURCES += sha1dc/%\n\n+UNIT_TEST_PROGRAMS += t-basic\n+UNIT_TEST_PROGRAMS += t-strbuf\n+UNIT_TEST_PROGS = $(patsubst %,$(UNIT_TEST_DIR)/%$X,$(UNIT_TEST_PROGRAMS))\n+UNIT_TEST_OBJS = $(patsubst %,$(UNIT_TEST_DIR)/%.o,$(UNIT_TEST_PROGRAMS))\n+UNIT_TEST_OBJS += $(UNIT_TEST_DIR)/test-lib.o\n+\n # xdiff and reftable libs may in turn depend on what is in libgit.a\n GITLIBS = common-main.o $(LIB_FILE) $(XDIFF_LIB) $(REFTABLE_LIB) $(LIB_FILE)\n EXTLIBS =\n@@ -2672,6 +2680,7 @@ OBJECTS += $(TEST_OBJS)\n OBJECTS += $(XDIFF_OBJS)\n OBJECTS += $(FUZZ_OBJS)\n OBJECTS += $(REFTABLE_OBJS) $(REFTABLE_TEST_OBJS)\n+OBJECTS += $(UNIT_TEST_OBJS)\n\n ifndef NO_CURL\n \tOBJECTS += http.o http-walker.o remote-curl.o\n@@ -3167,7 +3176,7 @@ endif\n\n test_bindir_programs := $(patsubst %,bin-wrappers/%,$(BINDIR_PROGRAMS_NEED_X) $(BINDIR_PROGRAMS_NO_X) $(TEST_PROGRAMS_NEED_X))\n\n-all:: $(TEST_PROGRAMS) $(test_bindir_programs)\n+all:: $(TEST_PROGRAMS) $(test_bindir_programs) $(UNIT_TEST_PROGS)\n\n bin-wrappers/%: wrap-for-bin.sh\n \t$(call mkdir_p_parent_template)\n@@ -3592,7 +3601,7 @@ endif\n\n artifacts-tar:: $(ALL_COMMANDS_TO_INSTALL) $(SCRIPT_LIB) $(OTHER_PROGRAMS) \\\n \t\tGIT-BUILD-OPTIONS $(TEST_PROGRAMS) $(test_bindir_programs) \\\n-\t\t$(MOFILES)\n+\t\t$(UNIT_TEST_PROGS) $(MOFILES)\n \t$(QUIET_SUBDIR0)templates $(QUIET_SUBDIR1) \\\n \t\tSHELL_PATH='$(SHELL_PATH_SQ)' PERL_PATH='$(PERL_PATH_SQ)'\n \ttest -n \"$(ARTIFACTS_DIRECTORY)\"\n@@ -3653,7 +3662,7 @@ clean: profile-clean coverage-clean cocciclean\n \t$(RM) $(OBJECTS)\n \t$(RM) $(LIB_FILE) $(XDIFF_LIB) $(REFTABLE_LIB) $(REFTABLE_TEST_LIB)\n \t$(RM) $(ALL_PROGRAMS) $(SCRIPT_LIB) $(BUILT_INS) $(OTHER_PROGRAMS)\n-\t$(RM) $(TEST_PROGRAMS)\n+\t$(RM) $(TEST_PROGRAMS) $(UNIT_TEST_PROGS)\n \t$(RM) $(FUZZ_PROGRAMS)\n \t$(RM) $(SP_OBJ)\n \t$(RM) $(HCC)\n@@ -3831,3 +3840,12 @@ $(FUZZ_PROGRAMS): all\n \t\t$(XDIFF_OBJS) $(EXTLIBS) git.o $@.o $(LIB_FUZZING_ENGINE) -o $@\n\n fuzz-all: $(FUZZ_PROGRAMS)\n+\n+$(UNIT_TEST_PROGS): $(UNIT_TEST_DIR)/%$X: $(UNIT_TEST_DIR)/%.o $(UNIT_TEST_DIR)/test-lib.o $(GITLIBS) GIT-LDFLAGS\n+\t$(QUIET_LINK)$(CC) $(ALL_CFLAGS) -o $@ $(ALL_LDFLAGS) \\\n+\t\t$(filter %.o,$^) $(filter %.a,$^) $(LIBS)\n+\n+.PHONY: build-unit-tests unit-tests\n+build-unit-tests: $(UNIT_TEST_PROGS)\n+unit-tests: $(UNIT_TEST_PROGS)\n+\t$(MAKE) -C t/ unit-tests\ndiff --git a/t/Makefile b/t/Makefile\nindex 3e00cdd801..92864cdf28 100644\n--- a/t/Makefile\n+++ b/t/Makefile\n@@ -41,6 +41,7 @@ TPERF = $(sort $(wildcard perf/p[0-9][0-9][0-9][0-9]-*.sh))\n TINTEROP = $(sort $(wildcard interop/i[0-9][0-9][0-9][0-9]-*.sh))\n CHAINLINTTESTS = $(sort $(patsubst chainlint/%.test,%,$(wildcard chainlint/*.test)))\n CHAINLINT = '$(PERL_PATH_SQ)' chainlint.pl\n+UNIT_TESTS = $(sort $(filter-out %.h %.c %.o unit-tests/t-basic%,$(wildcard unit-tests/*)))\n\n # `test-chainlint` (which is a dependency of `test-lint`, `test` and `prove`)\n # checks all tests in all scripts via a single invocation, so tell individual\n@@ -65,6 +66,13 @@ prove: pre-clean check-chainlint $(TEST_LINT)\n $(T):\n \t@echo \"*** $@ ***\"; '$(TEST_SHELL_PATH_SQ)' $@ $(GIT_TEST_OPTS)\n\n+$(UNIT_TESTS):\n+\t@echo \"*** $@ ***\"; $@\n+\n+.PHONY: unit-tests\n+unit-tests:\n+\t@echo \"*** prove - unit tests ***\"; $(PROVE) $(GIT_PROVE_OPTS) $(UNIT_TESTS)\n+\n pre-clean:\n \t$(RM) -r '$(TEST_RESULTS_DIRECTORY_SQ)'\n\n@@ -149,4 +157,4 @@ perf:\n \t$(MAKE) -C perf/ all\n\n .PHONY: pre-clean $(T) aggregate-results clean valgrind perf \\\n-\tcheck-chainlint clean-chainlint test-chainlint\n+\tcheck-chainlint clean-chainlint test-chainlint $(UNIT_TESTS)\ndiff --git a/t/t0080-unit-test-output.sh b/t/t0080-unit-test-output.sh\nnew file mode 100755\nindex 0000000000..c60e402260\n--- /dev/null\n+++ b/t/t0080-unit-test-output.sh\n@@ -0,0 +1,58 @@\n+#!/bin/sh\n+\n+test_description='Test the output of the unit test framework'\n+\n+. ./test-lib.sh\n+\n+test_expect_success 'TAP output from unit tests' '\n+\tcat >expect <<-EOF &&\n+\tok 1 - passing test\n+\tok 2 - passing test and assertion return 0\n+\t# check \"1 == 2\" failed at t/unit-tests/t-basic.c:68\n+\t# left: 1\n+\t# right: 2\n+\tnot ok 3 - failing test\n+\tok 4 - failing test and assertion return -1\n+\tnot ok 5 - passing TEST_TODO() # TODO\n+\tok 6 - passing TEST_TODO() returns 0\n+\t# todo check ${SQ}check(x)${SQ} succeeded at t/unit-tests/t-basic.c:17\n+\tnot ok 7 - failing TEST_TODO()\n+\tok 8 - failing TEST_TODO() returns -1\n+\t# check \"0\" failed at t/unit-tests/t-basic.c:22\n+\t# skipping test - missing prerequisite\n+\t# skipping check ${SQ}1${SQ} at t/unit-tests/t-basic.c:24\n+\tok 9 - test_skip() # SKIP\n+\tok 10 - skipped test returns 0\n+\t# skipping test - missing prerequisite\n+\tok 11 - test_skip() inside TEST_TODO() # SKIP\n+\tok 12 - test_skip() inside TEST_TODO() returns 0\n+\t# check \"0\" failed at t/unit-tests/t-basic.c:40\n+\tnot ok 13 - TEST_TODO() after failing check\n+\tok 14 - TEST_TODO() after failing check returns -1\n+\t# check \"0\" failed at t/unit-tests/t-basic.c:48\n+\tnot ok 15 - failing check after TEST_TODO()\n+\tok 16 - failing check after TEST_TODO() returns -1\n+\t# check \"!strcmp(\"\\thello\\\\\\\\\", \"there\\\"\\n\")\" failed at t/unit-tests/t-basic.c:53\n+\t# left: \"\\011hello\\\\\\\\\"\n+\t# right: \"there\\\"\\012\"\n+\t# check \"!strcmp(\"NULL\", NULL)\" failed at t/unit-tests/t-basic.c:54\n+\t# left: \"NULL\"\n+\t# right: NULL\n+\t# check \"${SQ}a${SQ} == ${SQ}\\n${SQ}\" failed at t/unit-tests/t-basic.c:55\n+\t# left: ${SQ}a${SQ}\n+\t# right: ${SQ}\\012${SQ}\n+\t# check \"${SQ}\\\\\\\\${SQ} == ${SQ}\\\\${SQ}${SQ}\" failed at t/unit-tests/t-basic.c:56\n+\t# left: ${SQ}\\\\\\\\${SQ}\n+\t# right: ${SQ}\\\\${SQ}${SQ}\n+\tnot ok 17 - messages from failing string and char comparison\n+\t# BUG: test has no checks at t/unit-tests/t-basic.c:83\n+\tnot ok 18 - test with no checks\n+\tok 19 - test with no checks returns -1\n+\t1..19\n+\tEOF\n+\n+\t! \"$GIT_BUILD_DIR\"/t/unit-tests/t-basic >actual &&\n+\ttest_cmp expect actual\n+'\n+\n+test_done\ndiff --git a/t/unit-tests/.gitignore b/t/unit-tests/.gitignore\nnew file mode 100644\nindex 0000000000..e292d58348\n--- /dev/null\n+++ b/t/unit-tests/.gitignore\n@@ -0,0 +1,2 @@\n+/t-basic\n+/t-strbuf\ndiff --git a/t/unit-tests/t-basic.c b/t/unit-tests/t-basic.c\nnew file mode 100644\nindex 0000000000..ab0b7682c4\n--- /dev/null\n+++ b/t/unit-tests/t-basic.c\n@@ -0,0 +1,87 @@\n+#include \"test-lib.h\"\n+\n+/* Used to store the return value of check_int(). */\n+static int check_res;\n+\n+/* Used to store the return value of TEST(). */\n+static int test_res;\n+\n+static void t_res(int expect)\n+{\n+\tcheck_int(check_res, ==, expect);\n+\tcheck_int(test_res, ==, expect);\n+}\n+\n+static void t_todo(int x)\n+{\n+\tcheck_res = TEST_TODO(check(x));\n+}\n+\n+static void t_skip(void)\n+{\n+\tcheck(0);\n+\ttest_skip(\"missing prerequisite\");\n+\tcheck(1);\n+}\n+\n+static int do_skip(void)\n+{\n+\ttest_skip(\"missing prerequisite\");\n+\treturn 0;\n+}\n+\n+static void t_skip_todo(void)\n+{\n+\tcheck_res = TEST_TODO(do_skip());\n+}\n+\n+static void t_todo_after_fail(void)\n+{\n+\tcheck(0);\n+\tTEST_TODO(check(0));\n+}\n+\n+static void t_fail_after_todo(void)\n+{\n+\tcheck(1);\n+\tTEST_TODO(check(0));\n+\tcheck(0);\n+}\n+\n+static void t_messages(void)\n+{\n+\tcheck_str(\"\\thello\\\\\", \"there\\\"\\n\");\n+\tcheck_str(\"NULL\", NULL);\n+\tcheck_char('a', ==, '\\n');\n+\tcheck_char('\\\\', ==, '\\'');\n+}\n+\n+static void t_empty(void)\n+{\n+\t; /* empty */\n+}\n+\n+int cmd_main(int argc, const char **argv)\n+{\n+\ttest_res = TEST(check_res = check_int(1, ==, 1), \"passing test\");\n+\tTEST(t_res(0), \"passing test and assertion return 0\");\n+\ttest_res = TEST(check_res = check_int(1, ==, 2), \"failing test\");\n+\tTEST(t_res(-1), \"failing test and assertion return -1\");\n+\ttest_res = TEST(t_todo(0), \"passing TEST_TODO()\");\n+\tTEST(t_res(0), \"passing TEST_TODO() returns 0\");\n+\ttest_res = TEST(t_todo(1), \"failing TEST_TODO()\");\n+\tTEST(t_res(-1), \"failing TEST_TODO() returns -1\");\n+\ttest_res = TEST(t_skip(), \"test_skip()\");\n+\tTEST(check_int(test_res, ==, 0), \"skipped test returns 0\");\n+\ttest_res = TEST(t_skip_todo(), \"test_skip() inside TEST_TODO()\");\n+\tTEST(t_res(0), \"test_skip() inside TEST_TODO() returns 0\");\n+\ttest_res = TEST(t_todo_after_fail(), \"TEST_TODO() after failing check\");\n+\tTEST(check_int(test_res, ==, -1), \"TEST_TODO() after failing check returns -1\");\n+\ttest_res = TEST(t_fail_after_todo(), \"failing check after TEST_TODO()\");\n+\tTEST(check_int(test_res, ==, -1), \"failing check after TEST_TODO() returns -1\");\n+\tTEST(t_messages(), \"messages from failing string and char comparison\");\n+\ttest_res = TEST(t_empty(), \"test with no checks\");\n+\tTEST(check_int(test_res, ==, -1), \"test with no checks returns -1\");\n+\n+\treturn test_done();\n+}\ndiff --git a/t/unit-tests/t-strbuf.c b/t/unit-tests/t-strbuf.c\nnew file mode 100644\nindex 0000000000..561611e242\n--- /dev/null\n+++ b/t/unit-tests/t-strbuf.c\n@@ -0,0 +1,75 @@\n+#include \"test-lib.h\"\n+#include \"strbuf.h\"\n+\n+/* wrapper that supplies tests with an initialized strbuf */\n+static void setup(void (*f)(struct strbuf*, void*), void *data)\n+{\n+\tstruct strbuf buf = STRBUF_INIT;\n+\n+\tf(&buf, data);\n+\tstrbuf_release(&buf);\n+\tcheck_uint(buf.len, ==, 0);\n+\tcheck_uint(buf.alloc, ==, 0);\n+\tcheck(buf.buf == strbuf_slopbuf);\n+\tcheck_char(buf.buf[0], ==, '\\0');\n+}\n+\n+static void t_static_init(void)\n+{\n+\tstruct strbuf buf = STRBUF_INIT;\n+\n+\tcheck_uint(buf.len, ==, 0);\n+\tcheck_uint(buf.alloc, ==, 0);\n+\tif (check(buf.buf == strbuf_slopbuf))\n+\t\treturn; /* avoid de-referencing buf.buf */\n+\tcheck_char(buf.buf[0], ==, '\\0');\n+}\n+\n+static void t_dynamic_init(void)\n+{\n+\tstruct strbuf buf;\n+\n+\tstrbuf_init(&buf, 1024);\n+\tcheck_uint(buf.len, ==, 0);\n+\tcheck_uint(buf.alloc, >=, 1024);\n+\tcheck_char(buf.buf[0], ==, '\\0');\n+\tstrbuf_release(&buf);\n+}\n+\n+static void t_addch(struct strbuf *buf, void *data)\n+{\n+\tconst char *p_ch = data;\n+\tconst char ch = *p_ch;\n+\n+\tstrbuf_addch(buf, ch);\n+\tif (check_uint(buf->len, ==, 1) ||\n+\t check_uint(buf->alloc, >, 1))\n+\t\treturn; /* avoid de-referencing buf->buf */\n+\tcheck_char(buf->buf[0], ==, ch);\n+\tcheck_char(buf->buf[1], ==, '\\0');\n+}\n+\n+static void t_addstr(struct strbuf *buf, void *data)\n+{\n+\tconst char *text = data;\n+\tsize_t len = strlen(text);\n+\n+\tstrbuf_addstr(buf, text);\n+\tif (check_uint(buf->len, ==, len) ||\n+\t check_uint(buf->alloc, >, len) ||\n+\t check_char(buf->buf[len], ==, '\\0'))\n+\t return;\n+\tcheck_str(buf->buf, text);\n+}\n+\n+int cmd_main(int argc, const char **argv)\n+{\n+\tif (TEST(t_static_init(), \"static initialization works\"))\n+\t\ttest_skip_all(\"STRBUF_INIT is broken\");\n+\tTEST(t_dynamic_init(), \"dynamic initialization works\");\n+\tTEST(setup(t_addch, \"a\"), \"strbuf_addch adds char\");\n+\tTEST(setup(t_addch, \"\"), \"strbuf_addch adds NUL char\");\n+\tTEST(setup(t_addstr, \"hello there\"), \"strbuf_addstr adds string\");\n+\n+\treturn test_done();\n+}\ndiff --git a/t/unit-tests/test-lib.c b/t/unit-tests/test-lib.c\nnew file mode 100644\nindex 0000000000..70030d587f\n--- /dev/null\n+++ b/t/unit-tests/test-lib.c\n@@ -0,0 +1,329 @@\n+#include \"test-lib.h\"\n+\n+enum result {\n+\tRESULT_NONE,\n+\tRESULT_FAILURE,\n+\tRESULT_SKIP,\n+\tRESULT_SUCCESS,\n+\tRESULT_TODO\n+};\n+\n+static struct {\n+\tenum result result;\n+\tint count;\n+\tunsigned failed :1;\n+\tunsigned lazy_plan :1;\n+\tunsigned running :1;\n+\tunsigned skip_all :1;\n+\tunsigned todo :1;\n+} ctx = {\n+\t.lazy_plan = 1,\n+\t.result = RESULT_NONE,\n+};\n+\n+static void msg_with_prefix(const char *prefix, const char *format, va_list ap)\n+{\n+\tfflush(stderr);\n+\tif (prefix)\n+\t\tfprintf(stdout, \"%s\", prefix);\n+\tvprintf(format, ap); /* TODO: handle newlines */\n+\tputc('\\n', stdout);\n+\tfflush(stdout);\n+}\n+\n+void test_msg(const char *format, ...)\n+{\n+\tva_list ap;\n+\n+\tva_start(ap, format);\n+\tmsg_with_prefix(\"# \", format, ap);\n+\tva_end(ap);\n+}\n+\n+void test_plan(int count)\n+{\n+\tassert(!ctx.running);\n+\n+\tfflush(stderr);\n+\tprintf(\"1..%d\\n\", count);\n+\tfflush(stdout);\n+\tctx.lazy_plan = 0;\n+}\n+\n+int test_done(void)\n+{\n+\tassert(!ctx.running);\n+\n+\tif (ctx.lazy_plan)\n+\t\ttest_plan(ctx.count);\n+\n+\treturn ctx.failed;\n+}\n+\n+void test_skip(const char *format, ...)\n+{\n+\tva_list ap;\n+\n+\tassert(ctx.running);\n+\n+\tctx.result = RESULT_SKIP;\n+\tva_start(ap, format);\n+\tif (format)\n+\t\tmsg_with_prefix(\"# skipping test - \", format, ap);\n+\tva_end(ap);\n+}\n+\n+void test_skip_all(const char *format, ...)\n+{\n+\tva_list ap;\n+\tconst char *prefix;\n+\n+\tif (!ctx.count && ctx.lazy_plan) {\n+\t\t/* We have not printed a test plan yet */\n+\t\tprefix = \"1..0 # SKIP \";\n+\t\tctx.lazy_plan = 0;\n+\t} else {\n+\t\t/* We have already printed a test plan */\n+\t\tprefix = \"Bail out! # \";\n+\t\tctx.failed = 1;\n+\t}\n+\tctx.skip_all = 1;\n+\tctx.result = RESULT_SKIP;\n+\tva_start(ap, format);\n+\tmsg_with_prefix(prefix, format, ap);\n+\tva_end(ap);\n+}\n+\n+int test__run_begin(void)\n+{\n+\tassert(!ctx.running);\n+\n+\tctx.count++;\n+\tctx.result = RESULT_NONE;\n+\tctx.running = 1;\n+\n+\treturn ctx.skip_all;\n+}\n+\n+static void print_description(const char *format, va_list ap)\n+{\n+\tif (format) {\n+\t\tfputs(\" - \", stdout);\n+\t\tvprintf(format, ap);\n+\t}\n+}\n+\n+int test__run_end(int was_run UNUSED, const char *location, const char *format, ...)\n+{\n+\tva_list ap;\n+\n+\tassert(ctx.running);\n+\tassert(!ctx.todo);\n+\n+\tfflush(stderr);\n+\tva_start(ap, format);\n+\tif (!ctx.skip_all) {\n+\t\tswitch (ctx.result) {\n+\t\tcase RESULT_SUCCESS:\n+\t\t\tprintf(\"ok %d\", ctx.count);\n+\t\t\tprint_description(format, ap);\n+\t\t\tbreak;\n+\n+\t\tcase RESULT_FAILURE:\n+\t\t\tprintf(\"not ok %d\", ctx.count);\n+\t\t\tprint_description(format, ap);\n+\t\t\tbreak;\n+\n+\t\tcase RESULT_TODO:\n+\t\t\tprintf(\"not ok %d\", ctx.count);\n+\t\t\tprint_description(format, ap);\n+\t\t\tprintf(\" # TODO\");\n+\t\t\tbreak;\n+\n+\t\tcase RESULT_SKIP:\n+\t\t\tprintf(\"ok %d\", ctx.count);\n+\t\t\tprint_description(format, ap);\n+\t\t\tprintf(\" # SKIP\");\n+\t\t\tbreak;\n+\n+\t\tcase RESULT_NONE:\n+\t\t\ttest_msg(\"BUG: test has no checks at %s\", location);\n+\t\t\tprintf(\"not ok %d\", ctx.count);\n+\t\t\tprint_description(format, ap);\n+\t\t\tctx.result = RESULT_FAILURE;\n+\t\t\tbreak;\n+\t\t}\n+\t}\n+\tva_end(ap);\n+\tctx.running = 0;\n+\tif (ctx.skip_all)\n+\t\treturn 0;\n+\tputc('\\n', stdout);\n+\tfflush(stdout);\n+\tctx.failed |= ctx.result == RESULT_FAILURE;\n+\n+\treturn -(ctx.result == RESULT_FAILURE);\n+}\n+\n+static void test_fail(void)\n+{\n+\tassert(ctx.result != RESULT_SKIP);\n+\n+\tctx.result = RESULT_FAILURE;\n+}\n+\n+static void test_pass(void)\n+{\n+\tassert(ctx.result != RESULT_SKIP);\n+\n+\tif (ctx.result == RESULT_NONE)\n+\t\tctx.result = RESULT_SUCCESS;\n+}\n+\n+static void test_todo(void)\n+{\n+\tassert(ctx.result != RESULT_SKIP);\n+\n+\tif (ctx.result != RESULT_FAILURE)\n+\t\tctx.result = RESULT_TODO;\n+}\n+\n+int test_assert(const char *location, const char *check, int ok)\n+{\n+\tassert(ctx.running);\n+\n+\tif (ctx.result == RESULT_SKIP) {\n+\t\ttest_msg(\"skipping check '%s' at %s\", check, location);\n+\t\treturn 0;\n+\t} else if (!ctx.todo) {\n+\t\tif (ok) {\n+\t\t\ttest_pass();\n+\t\t} else {\n+\t\t\ttest_msg(\"check \\\"%s\\\" failed at %s\", check, location);\n+\t\t\ttest_fail();\n+\t\t}\n+\t}\n+\n+\treturn -!ok;\n+}\n+\n+void test__todo_begin(void)\n+{\n+\tassert(ctx.running);\n+\tassert(!ctx.todo);\n+\n+\tctx.todo = 1;\n+}\n+\n+int test__todo_end(const char *location, const char *check, int res)\n+{\n+\tassert(ctx.running);\n+\tassert(ctx.todo);\n+\n+\tctx.todo = 0;\n+\tif (ctx.result == RESULT_SKIP)\n+\t\treturn 0;\n+\tif (!res) {\n+\t\ttest_msg(\"todo check '%s' succeeded at %s\", check, location);\n+\t\ttest_fail();\n+\t} else {\n+\t\ttest_todo();\n+\t}\n+\n+\treturn -!res;\n+}\n+\n+int check_bool_loc(const char *loc, const char *check, int ok)\n+{\n+\treturn test_assert(loc, check, ok);\n+}\n+\n+union test__tmp test__tmp[2];\n+\n+int check_int_loc(const char *loc, const char *check, int ok,\n+\t\t intmax_t a, intmax_t b)\n+{\n+\tint ret = test_assert(loc, check, ok);\n+\n+\tif (ret) {\n+\t\ttest_msg(\" left: %\"PRIdMAX, a);\n+\t\ttest_msg(\" right: %\"PRIdMAX, b);\n+\t}\n+\n+\treturn ret;\n+}\n+\n+int check_uint_loc(const char *loc, const char *check, int ok,\n+\t\t uintmax_t a, uintmax_t b)\n+{\n+\tint ret = test_assert(loc, check, ok);\n+\n+\tif (ret) {\n+\t\ttest_msg(\" left: %\"PRIuMAX, a);\n+\t\ttest_msg(\" right: %\"PRIuMAX, b);\n+\t}\n+\n+\treturn ret;\n+}\n+\n+static void print_one_char(char ch, char quote)\n+{\n+\tif ((unsigned char)ch < 0x20u || ch == 0x7f) {\n+\t\t/* TODO: improve handling of \\a, \\b, \\f ... */\n+\t\tprintf(\"\\\\%03o\", (unsigned char)ch);\n+\t} else {\n+\t\tif (ch == '\\\\' || ch == quote)\n+\t\t\tputc('\\\\', stdout);\n+\t\tputc(ch, stdout);\n+\t}\n+}\n+\n+static void print_char(const char *prefix, char ch)\n+{\n+\tprintf(\"# %s: '\", prefix);\n+\tprint_one_char(ch, '\\'');\n+\tfputs(\"'\\n\", stdout);\n+}\n+\n+int check_char_loc(const char *loc, const char *check, int ok, char a, char b)\n+{\n+\tint ret = test_assert(loc, check, ok);\n+\n+\tif (ret) {\n+\t\tfflush(stderr);\n+\t\tprint_char(\" left\", a);\n+\t\tprint_char(\" right\", b);\n+\t\tfflush(stdout);\n+\t}\n+\n+\treturn ret;\n+}\n+\n+static void print_str(const char *prefix, const char *str)\n+{\n+\tprintf(\"# %s: \", prefix);\n+\tif (!str) {\n+\t\tfputs(\"NULL\\n\", stdout);\n+\t} else {\n+\t\tputc('\"', stdout);\n+\t\twhile (*str)\n+\t\t\tprint_one_char(*str++, '\"');\n+\t\tfputs(\"\\\"\\n\", stdout);\n+\t}\n+}\n+\n+int check_str_loc(const char *loc, const char *check,\n+\t\t const char *a, const char *b)\n+{\n+\tint ok = (!a && !b) || (a && b && !strcmp(a, b));\n+\tint ret = test_assert(loc, check, ok);\n+\n+\tif (ret) {\n+\t\tfflush(stderr);\n+\t\tprint_str(\" left\", a);\n+\t\tprint_str(\" right\", b);\n+\t\tfflush(stdout);\n+\t}\n+\n+\treturn ret;\n+}\ndiff --git a/t/unit-tests/test-lib.h b/t/unit-tests/test-lib.h\nnew file mode 100644\nindex 0000000000..720c97c6f8\n--- /dev/null\n+++ b/t/unit-tests/test-lib.h\n@@ -0,0 +1,143 @@\n+#ifndef TEST_LIB_H\n+#define TEST_LIB_H\n+\n+#include \"git-compat-util.h\"\n+\n+/*\n+ * Run a test function, returns 0 if the test succeeds, -1 if it\n+ * fails. If test_skip_all() has been called then the test will not be\n+ * run. The description for each test should be unique. For example:\n+ *\n+ * TEST(test_something(arg1, arg2), \"something %d %d\", arg1, arg2)\n+ */\n+#define TEST(t, ...)\t\t\t\t\t\\\n+\ttest__run_end(test__run_begin() ? 0 : (t, 1),\t\\\n+\t\t TEST_LOCATION(), __VA_ARGS__)\n+\n+/*\n+ * Print a test plan, should be called before any tests. If the number\n+ * of tests is not known in advance test_done() will automatically\n+ * print a plan at the end of the test program.\n+ */\n+void test_plan(int count);\n+\n+/*\n+ * test_done() must be called at the end of main(). It will print the\n+ * plan if plan() was not called at the beginning of the test program\n+ * and returns the exit code for the test program.\n+ */\n+int test_done(void);\n+\n+/* Skip the current test. */\n+__attribute__((format (printf, 1, 2)))\n+void test_skip(const char *format, ...);\n+\n+/* Skip all remaining tests. */\n+__attribute__((format (printf, 1, 2)))\n+void test_skip_all(const char *format, ...);\n+\n+/* Print a diagnostic message to stdout. */\n+__attribute__((format (printf, 1, 2)))\n+void test_msg(const char *format, ...);\n+\n+/*\n+ * Test checks are built around test_assert(). checks return 0 on\n+ * success, -1 on failure. If any check fails then the test will\n+ * fail. To create a custom check define a function that wraps\n+ * test_assert() and a macro to wrap that function. For example:\n+ *\n+ * static int check_oid_loc(const char *loc, const char *check,\n+ *\t\t\t struct object_id *a, struct object_id *b)\n+ * {\n+ *\t int res = test_assert(loc, check, oideq(a, b));\n+ *\n+ *\t if (res) {\n+ *\t\t test_msg(\" left: %s\", oid_to_hex(a);\n+ *\t\t test_msg(\" right: %s\", oid_to_hex(a);\n+ *\n+ *\t }\n+ *\t return res;\n+ * }\n+ *\n+ * #define check_oid(a, b) \\\n+ *\t check_oid_loc(TEST_LOCATION(), \"oideq(\"#a\", \"#b\")\", a, b)\n+ */\n+int test_assert(const char *location, const char *check, int ok);\n+\n+/* Helper macro to pass the location to checks */\n+#define TEST_LOCATION() TEST__MAKE_LOCATION(__LINE__)\n+\n+/* Check a boolean condition. */\n+#define check(x)\t\t\t\t\\\n+\tcheck_bool_loc(TEST_LOCATION(), #x, x)\n+int check_bool_loc(const char *loc, const char *check, int ok);\n+\n+/*\n+ * Compare two integers. Prints a message with the two values if the\n+ * comparison fails. NB this is not thread safe.\n+ */\n+#define check_int(a, op, b)\t\t\t\t\t\t\\\n+\t(test__tmp[0].i = (a), test__tmp[1].i = (b),\t\t\t\\\n+\t check_int_loc(TEST_LOCATION(), #a\" \"#op\" \"#b,\t\t\t\\\n+\t\t test__tmp[0].i op test__tmp[1].i, a, b))\n+int check_int_loc(const char *loc, const char *check, int ok,\n+\t\t intmax_t a, intmax_t b);\n+\n+/*\n+ * Compare two unsigned integers. Prints a message with the two values\n+ * if the comparison fails. NB this is not thread safe.\n+ */\n+#define check_uint(a, op, b)\t\t\t\t\t\t\\\n+\t(test__tmp[0].u = (a), test__tmp[1].u = (b),\t\t\t\\\n+\t check_uint_loc(TEST_LOCATION(), #a\" \"#op\" \"#b,\t\t\t\\\n+\t\t\ttest__tmp[0].u op test__tmp[1].u, a, b))\n+int check_uint_loc(const char *loc, const char *check, int ok,\n+\t\t uintmax_t a, uintmax_t b);\n+\n+/*\n+ * Compare two chars. Prints a message with the two values if the\n+ * comparison fails. NB this is not thread safe.\n+ */\n+#define check_char(a, op, b)\t\t\t\t\t\t\\\n+\t(test__tmp[0].c = (a), test__tmp[1].c = (b),\t\t\t\\\n+\t check_char_loc(TEST_LOCATION(), #a\" \"#op\" \"#b,\t\t\t\\\n+\t\t\ttest__tmp[0].c op test__tmp[1].c, a, b))\n+int check_char_loc(const char *loc, const char *check, int ok,\n+\t\t char a, char b);\n+\n+/* Check whether two strings are equal. */\n+#define check_str(a, b)\t\t\t\t\t\t\t\\\n+\tcheck_str_loc(TEST_LOCATION(), \"!strcmp(\"#a\", \"#b\")\", a, b)\n+int check_str_loc(const char *loc, const char *check,\n+\t\t const char *a, const char *b);\n+\n+/*\n+ * Wrap a check that is known to fail. If the check succeeds then the\n+ * test will fail. Returns 0 if the check fails, -1 if it\n+ * succeeds. For example:\n+ *\n+ * TEST_TODO(check(0));\n+ */\n+#define TEST_TODO(check) \\\n+\t(test__todo_begin(), test__todo_end(TEST_LOCATION(), #check, check))\n+\n+/* Private helpers */\n+\n+#define TEST__STR(x) #x\n+#define TEST__MAKE_LOCATION(line) __FILE__ \":\" TEST__STR(line)\n+\n+union test__tmp {\n+\tintmax_t i;\n+\tuintmax_t u;\n+\tchar c;\n+};\n+\n+extern union test__tmp test__tmp[2];\n+\n+int test__run_begin(void);\n+__attribute__((format (printf, 3, 4)))\n+int test__run_end(int, const char *, const char *, ...);\n+void test__todo_begin(void);\n+int test__todo_end(const char *, const char *, int);\n+\n+#endif /* TEST_LIB_H */\n\nSigned-off-by: Josh Steadmon \n---\n Makefile | 24 ++-\n t/Makefile | 15 +-\n t/t0080-unit-test-output.sh | 58 +++++++\n t/unit-tests/.gitignore | 2 +\n t/unit-tests/t-basic.c | 95 +++++++++++\n t/unit-tests/t-strbuf.c | 75 ++++++++\n t/unit-tests/test-lib.c | 329 ++++++++++++++++++++++++++++++++++++\n t/unit-tests/test-lib.h | 143 ++++++++++++++++\n 8 files changed, 737 insertions(+), 4 deletions(-)\n create mode 100755 t/t0080-unit-test-output.sh\n create mode 100644 t/unit-tests/.gitignore\n create mode 100644 t/unit-tests/t-basic.c\n create mode 100644 t/unit-tests/t-strbuf.c\n create mode 100644 t/unit-tests/test-lib.c\n create mode 100644 t/unit-tests/test-lib.h\n\ndiff --git a/Makefile b/Makefile\nindex e440728c24..4016da6e39 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -682,6 +682,8 @@ TEST_BUILTINS_OBJS =\n TEST_OBJS =\n TEST_PROGRAMS_NEED_X =\n THIRD_PARTY_SOURCES =\n+UNIT_TEST_PROGRAMS =\n+UNIT_TEST_DIR = t/unit-tests\n \n # Having this variable in your environment would break pipelines because\n # you cause \"cd\" to echo its destination to stdout. It can also take\n@@ -1331,6 +1333,12 @@ THIRD_PARTY_SOURCES += compat/regex/%\n THIRD_PARTY_SOURCES += sha1collisiondetection/%\n THIRD_PARTY_SOURCES += sha1dc/%\n \n+UNIT_TEST_PROGRAMS += t-basic\n+UNIT_TEST_PROGRAMS += t-strbuf\n+UNIT_TEST_PROGS = $(patsubst %,$(UNIT_TEST_DIR)/%$X,$(UNIT_TEST_PROGRAMS))\n+UNIT_TEST_OBJS = $(patsubst %,$(UNIT_TEST_DIR)/%.o,$(UNIT_TEST_PROGRAMS))\n+UNIT_TEST_OBJS += $(UNIT_TEST_DIR)/test-lib.o\n+\n # xdiff and reftable libs may in turn depend on what is in libgit.a\n GITLIBS = common-main.o $(LIB_FILE) $(XDIFF_LIB) $(REFTABLE_LIB) $(LIB_FILE)\n EXTLIBS =\n@@ -2672,6 +2680,7 @@ OBJECTS += $(TEST_OBJS)\n OBJECTS += $(XDIFF_OBJS)\n OBJECTS += $(FUZZ_OBJS)\n OBJECTS += $(REFTABLE_OBJS) $(REFTABLE_TEST_OBJS)\n+OBJECTS += $(UNIT_TEST_OBJS)\n \n ifndef NO_CURL\n \tOBJECTS += http.o http-walker.o remote-curl.o\n@@ -3167,7 +3176,7 @@ endif\n \n test_bindir_programs := $(patsubst %,bin-wrappers/%,$(BINDIR_PROGRAMS_NEED_X) $(BINDIR_PROGRAMS_NO_X) $(TEST_PROGRAMS_NEED_X))\n \n-all:: $(TEST_PROGRAMS) $(test_bindir_programs)\n+all:: $(TEST_PROGRAMS) $(test_bindir_programs) $(UNIT_TEST_PROGS)\n \n bin-wrappers/%: wrap-for-bin.sh\n \t$(call mkdir_p_parent_template)\n@@ -3592,7 +3601,7 @@ endif\n \n artifacts-tar:: $(ALL_COMMANDS_TO_INSTALL) $(SCRIPT_LIB) $(OTHER_PROGRAMS) \\\n \t\tGIT-BUILD-OPTIONS $(TEST_PROGRAMS) $(test_bindir_programs) \\\n-\t\t$(MOFILES)\n+\t\t$(UNIT_TEST_PROGS) $(MOFILES)\n \t$(QUIET_SUBDIR0)templates $(QUIET_SUBDIR1) \\\n \t\tSHELL_PATH='$(SHELL_PATH_SQ)' PERL_PATH='$(PERL_PATH_SQ)'\n \ttest -n \"$(ARTIFACTS_DIRECTORY)\"\n@@ -3653,7 +3662,7 @@ clean: profile-clean coverage-clean cocciclean\n \t$(RM) $(OBJECTS)\n \t$(RM) $(LIB_FILE) $(XDIFF_LIB) $(REFTABLE_LIB) $(REFTABLE_TEST_LIB)\n \t$(RM) $(ALL_PROGRAMS) $(SCRIPT_LIB) $(BUILT_INS) $(OTHER_PROGRAMS)\n-\t$(RM) $(TEST_PROGRAMS)\n+\t$(RM) $(TEST_PROGRAMS) $(UNIT_TEST_PROGS)\n \t$(RM) $(FUZZ_PROGRAMS)\n \t$(RM) $(SP_OBJ)\n \t$(RM) $(HCC)\n@@ -3831,3 +3840,12 @@ $(FUZZ_PROGRAMS): all\n \t\t$(XDIFF_OBJS) $(EXTLIBS) git.o $@.o $(LIB_FUZZING_ENGINE) -o $@\n \n fuzz-all: $(FUZZ_PROGRAMS)\n+\n+$(UNIT_TEST_PROGS): $(UNIT_TEST_DIR)/%$X: $(UNIT_TEST_DIR)/%.o $(UNIT_TEST_DIR)/test-lib.o $(GITLIBS) GIT-LDFLAGS\n+\t$(QUIET_LINK)$(CC) $(ALL_CFLAGS) -o $@ $(ALL_LDFLAGS) \\\n+\t\t$(filter %.o,$^) $(filter %.a,$^) $(LIBS)\n+\n+.PHONY: build-unit-tests unit-tests\n+build-unit-tests: $(UNIT_TEST_PROGS)\n+unit-tests: $(UNIT_TEST_PROGS)\n+\t$(MAKE) -C t/ unit-tests\ndiff --git a/t/Makefile b/t/Makefile\nindex 3e00cdd801..2db8b3adb1 100644\n--- a/t/Makefile\n+++ b/t/Makefile\n@@ -17,6 +17,7 @@ TAR ?= $(TAR)\n RM ?= rm -f\n PROVE ?= prove\n DEFAULT_TEST_TARGET ?= test\n+DEFAULT_UNIT_TEST_TARGET ?= unit-tests-raw\n TEST_LINT ?= test-lint\n \n ifdef TEST_OUTPUT_DIRECTORY\n@@ -41,6 +42,7 @@ TPERF = $(sort $(wildcard perf/p[0-9][0-9][0-9][0-9]-*.sh))\n TINTEROP = $(sort $(wildcard interop/i[0-9][0-9][0-9][0-9]-*.sh))\n CHAINLINTTESTS = $(sort $(patsubst chainlint/%.test,%,$(wildcard chainlint/*.test)))\n CHAINLINT = '$(PERL_PATH_SQ)' chainlint.pl\n+UNIT_TESTS = $(sort $(filter-out %.h %.c %.o unit-tests/t-basic%,$(wildcard unit-tests/*)))\n \n # `test-chainlint` (which is a dependency of `test-lint`, `test` and `prove`)\n # checks all tests in all scripts via a single invocation, so tell individual\n@@ -65,6 +67,17 @@ prove: pre-clean check-chainlint $(TEST_LINT)\n $(T):\n \t@echo \"*** $@ ***\"; '$(TEST_SHELL_PATH_SQ)' $@ $(GIT_TEST_OPTS)\n \n+$(UNIT_TESTS):\n+\t@echo \"*** $@ ***\"; $@\n+\n+.PHONY: unit-tests unit-tests-raw unit-tests-prove\n+unit-tests: $(DEFAULT_UNIT_TEST_TARGET)\n+\n+unit-tests-raw: $(UNIT_TESTS)\n+\n+unit-tests-prove:\n+\t@echo \"*** prove - unit tests ***\"; $(PROVE) $(GIT_PROVE_OPTS) $(UNIT_TESTS)\n+\n pre-clean:\n \t$(RM) -r '$(TEST_RESULTS_DIRECTORY_SQ)'\n \n@@ -149,4 +162,4 @@ perf:\n \t$(MAKE) -C perf/ all\n \n .PHONY: pre-clean $(T) aggregate-results clean valgrind perf \\\n-\tcheck-chainlint clean-chainlint test-chainlint\n+\tcheck-chainlint clean-chainlint test-chainlint $(UNIT_TESTS)\ndiff --git a/t/t0080-unit-test-output.sh b/t/t0080-unit-test-output.sh\nnew file mode 100755\nindex 0000000000..e0fc07d1e5\n--- /dev/null\n+++ b/t/t0080-unit-test-output.sh\n@@ -0,0 +1,58 @@\n+#!/bin/sh\n+\n+test_description='Test the output of the unit test framework'\n+\n+. ./test-lib.sh\n+\n+test_expect_success 'TAP output from unit tests' '\n+\tcat >expect <<-EOF &&\n+\tok 1 - passing test\n+\tok 2 - passing test and assertion return 0\n+\t# check \"1 == 2\" failed at t/unit-tests/t-basic.c:76\n+\t# left: 1\n+\t# right: 2\n+\tnot ok 3 - failing test\n+\tok 4 - failing test and assertion return -1\n+\tnot ok 5 - passing TEST_TODO() # TODO\n+\tok 6 - passing TEST_TODO() returns 0\n+\t# todo check ${SQ}check(x)${SQ} succeeded at t/unit-tests/t-basic.c:25\n+\tnot ok 7 - failing TEST_TODO()\n+\tok 8 - failing TEST_TODO() returns -1\n+\t# check \"0\" failed at t/unit-tests/t-basic.c:30\n+\t# skipping test - missing prerequisite\n+\t# skipping check ${SQ}1${SQ} at t/unit-tests/t-basic.c:32\n+\tok 9 - test_skip() # SKIP\n+\tok 10 - skipped test returns 0\n+\t# skipping test - missing prerequisite\n+\tok 11 - test_skip() inside TEST_TODO() # SKIP\n+\tok 12 - test_skip() inside TEST_TODO() returns 0\n+\t# check \"0\" failed at t/unit-tests/t-basic.c:48\n+\tnot ok 13 - TEST_TODO() after failing check\n+\tok 14 - TEST_TODO() after failing check returns -1\n+\t# check \"0\" failed at t/unit-tests/t-basic.c:56\n+\tnot ok 15 - failing check after TEST_TODO()\n+\tok 16 - failing check after TEST_TODO() returns -1\n+\t# check \"!strcmp(\"\\thello\\\\\\\\\", \"there\\\"\\n\")\" failed at t/unit-tests/t-basic.c:61\n+\t# left: \"\\011hello\\\\\\\\\"\n+\t# right: \"there\\\"\\012\"\n+\t# check \"!strcmp(\"NULL\", NULL)\" failed at t/unit-tests/t-basic.c:62\n+\t# left: \"NULL\"\n+\t# right: NULL\n+\t# check \"${SQ}a${SQ} == ${SQ}\\n${SQ}\" failed at t/unit-tests/t-basic.c:63\n+\t# left: ${SQ}a${SQ}\n+\t# right: ${SQ}\\012${SQ}\n+\t# check \"${SQ}\\\\\\\\${SQ} == ${SQ}\\\\${SQ}${SQ}\" failed at t/unit-tests/t-basic.c:64\n+\t# left: ${SQ}\\\\\\\\${SQ}\n+\t# right: ${SQ}\\\\${SQ}${SQ}\n+\tnot ok 17 - messages from failing string and char comparison\n+\t# BUG: test has no checks at t/unit-tests/t-basic.c:91\n+\tnot ok 18 - test with no checks\n+\tok 19 - test with no checks returns -1\n+\t1..19\n+\tEOF\n+\n+\t! \"$GIT_BUILD_DIR\"/t/unit-tests/t-basic >actual &&\n+\ttest_cmp expect actual\n+'\n+\n+test_done\ndiff --git a/t/unit-tests/.gitignore b/t/unit-tests/.gitignore\nnew file mode 100644\nindex 0000000000..e292d58348\n--- /dev/null\n+++ b/t/unit-tests/.gitignore\n@@ -0,0 +1,2 @@\n+/t-basic\n+/t-strbuf\ndiff --git a/t/unit-tests/t-basic.c b/t/unit-tests/t-basic.c\nnew file mode 100644\nindex 0000000000..d20f444fab\n--- /dev/null\n+++ b/t/unit-tests/t-basic.c\n@@ -0,0 +1,95 @@\n+#include \"test-lib.h\"\n+\n+/*\n+ * The purpose of this \"unit test\" is to verify a few invariants of the unit\n+ * test framework itself, as well as to provide examples of output from actually\n+ * failing tests. As such, it is intended that this test fails, and thus it\n+ * should not be run as part of `make unit-tests`. Instead, we verify it behaves\n+ * as expected in the integration test t0080-unit-test-output.sh\n+ */\n+\n+/* Used to store the return value of check_int(). */\n+static int check_res;\n+\n+/* Used to store the return value of TEST(). */\n+static int test_res;\n+\n+static void t_res(int expect)\n+{\n+\tcheck_int(check_res, ==, expect);\n+\tcheck_int(test_res, ==, expect);\n+}\n+\n+static void t_todo(int x)\n+{\n+\tcheck_res = TEST_TODO(check(x));\n+}\n+\n+static void t_skip(void)\n+{\n+\tcheck(0);\n+\ttest_skip(\"missing prerequisite\");\n+\tcheck(1);\n+}\n+\n+static int do_skip(void)\n+{\n+\ttest_skip(\"missing prerequisite\");\n+\treturn 0;\n+}\n+\n+static void t_skip_todo(void)\n+{\n+\tcheck_res = TEST_TODO(do_skip());\n+}\n+\n+static void t_todo_after_fail(void)\n+{\n+\tcheck(0);\n+\tTEST_TODO(check(0));\n+}\n+\n+static void t_fail_after_todo(void)\n+{\n+\tcheck(1);\n+\tTEST_TODO(check(0));\n+\tcheck(0);\n+}\n+\n+static void t_messages(void)\n+{\n+\tcheck_str(\"\\thello\\\\\", \"there\\\"\\n\");\n+\tcheck_str(\"NULL\", NULL);\n+\tcheck_char('a', ==, '\\n');\n+\tcheck_char('\\\\', ==, '\\'');\n+}\n+\n+static void t_empty(void)\n+{\n+\t; /* empty */\n+}\n+\n+int cmd_main(int argc, const char **argv)\n+{\n+\ttest_res = TEST(check_res = check_int(1, ==, 1), \"passing test\");\n+\tTEST(t_res(0), \"passing test and assertion return 0\");\n+\ttest_res = TEST(check_res = check_int(1, ==, 2), \"failing test\");\n+\tTEST(t_res(-1), \"failing test and assertion return -1\");\n+\ttest_res = TEST(t_todo(0), \"passing TEST_TODO()\");\n+\tTEST(t_res(0), \"passing TEST_TODO() returns 0\");\n+\ttest_res = TEST(t_todo(1), \"failing TEST_TODO()\");\n+\tTEST(t_res(-1), \"failing TEST_TODO() returns -1\");\n+\ttest_res = TEST(t_skip(), \"test_skip()\");\n+\tTEST(check_int(test_res, ==, 0), \"skipped test returns 0\");\n+\ttest_res = TEST(t_skip_todo(), \"test_skip() inside TEST_TODO()\");\n+\tTEST(t_res(0), \"test_skip() inside TEST_TODO() returns 0\");\n+\ttest_res = TEST(t_todo_after_fail(), \"TEST_TODO() after failing check\");\n+\tTEST(check_int(test_res, ==, -1), \"TEST_TODO() after failing check returns -1\");\n+\ttest_res = TEST(t_fail_after_todo(), \"failing check after TEST_TODO()\");\n+\tTEST(check_int(test_res, ==, -1), \"failing check after TEST_TODO() returns -1\");\n+\tTEST(t_messages(), \"messages from failing string and char comparison\");\n+\ttest_res = TEST(t_empty(), \"test with no checks\");\n+\tTEST(check_int(test_res, ==, -1), \"test with no checks returns -1\");\n+\n+\treturn test_done();\n+}\ndiff --git a/t/unit-tests/t-strbuf.c b/t/unit-tests/t-strbuf.c\nnew file mode 100644\nindex 0000000000..561611e242\n--- /dev/null\n+++ b/t/unit-tests/t-strbuf.c\n@@ -0,0 +1,75 @@\n+#include \"test-lib.h\"\n+#include \"strbuf.h\"\n+\n+/* wrapper that supplies tests with an initialized strbuf */\n+static void setup(void (*f)(struct strbuf*, void*), void *data)\n+{\n+\tstruct strbuf buf = STRBUF_INIT;\n+\n+\tf(&buf, data);\n+\tstrbuf_release(&buf);\n+\tcheck_uint(buf.len, ==, 0);\n+\tcheck_uint(buf.alloc, ==, 0);\n+\tcheck(buf.buf == strbuf_slopbuf);\n+\tcheck_char(buf.buf[0], ==, '\\0');\n+}\n+\n+static void t_static_init(void)\n+{\n+\tstruct strbuf buf = STRBUF_INIT;\n+\n+\tcheck_uint(buf.len, ==, 0);\n+\tcheck_uint(buf.alloc, ==, 0);\n+\tif (check(buf.buf == strbuf_slopbuf))\n+\t\treturn; /* avoid de-referencing buf.buf */\n+\tcheck_char(buf.buf[0], ==, '\\0');\n+}\n+\n+static void t_dynamic_init(void)\n+{\n+\tstruct strbuf buf;\n+\n+\tstrbuf_init(&buf, 1024);\n+\tcheck_uint(buf.len, ==, 0);\n+\tcheck_uint(buf.alloc, >=, 1024);\n+\tcheck_char(buf.buf[0], ==, '\\0');\n+\tstrbuf_release(&buf);\n+}\n+\n+static void t_addch(struct strbuf *buf, void *data)\n+{\n+\tconst char *p_ch = data;\n+\tconst char ch = *p_ch;\n+\n+\tstrbuf_addch(buf, ch);\n+\tif (check_uint(buf->len, ==, 1) ||\n+\t check_uint(buf->alloc, >, 1))\n+\t\treturn; /* avoid de-referencing buf->buf */\n+\tcheck_char(buf->buf[0], ==, ch);\n+\tcheck_char(buf->buf[1], ==, '\\0');\n+}\n+\n+static void t_addstr(struct strbuf *buf, void *data)\n+{\n+\tconst char *text = data;\n+\tsize_t len = strlen(text);\n+\n+\tstrbuf_addstr(buf, text);\n+\tif (check_uint(buf->len, ==, len) ||\n+\t check_uint(buf->alloc, >, len) ||\n+\t check_char(buf->buf[len], ==, '\\0'))\n+\t return;\n+\tcheck_str(buf->buf, text);\n+}\n+\n+int cmd_main(int argc, const char **argv)\n+{\n+\tif (TEST(t_static_init(), \"static initialization works\"))\n+\t\ttest_skip_all(\"STRBUF_INIT is broken\");\n+\tTEST(t_dynamic_init(), \"dynamic initialization works\");\n+\tTEST(setup(t_addch, \"a\"), \"strbuf_addch adds char\");\n+\tTEST(setup(t_addch, \"\"), \"strbuf_addch adds NUL char\");\n+\tTEST(setup(t_addstr, \"hello there\"), \"strbuf_addstr adds string\");\n+\n+\treturn test_done();\n+}\ndiff --git a/t/unit-tests/test-lib.c b/t/unit-tests/test-lib.c\nnew file mode 100644\nindex 0000000000..70030d587f\n--- /dev/null\n+++ b/t/unit-tests/test-lib.c\n@@ -0,0 +1,329 @@\n+#include \"test-lib.h\"\n+\n+enum result {\n+\tRESULT_NONE,\n+\tRESULT_FAILURE,\n+\tRESULT_SKIP,\n+\tRESULT_SUCCESS,\n+\tRESULT_TODO\n+};\n+\n+static struct {\n+\tenum result result;\n+\tint count;\n+\tunsigned failed :1;\n+\tunsigned lazy_plan :1;\n+\tunsigned running :1;\n+\tunsigned skip_all :1;\n+\tunsigned todo :1;\n+} ctx = {\n+\t.lazy_plan = 1,\n+\t.result = RESULT_NONE,\n+};\n+\n+static void msg_with_prefix(const char *prefix, const char *format, va_list ap)\n+{\n+\tfflush(stderr);\n+\tif (prefix)\n+\t\tfprintf(stdout, \"%s\", prefix);\n+\tvprintf(format, ap); /* TODO: handle newlines */\n+\tputc('\\n', stdout);\n+\tfflush(stdout);\n+}\n+\n+void test_msg(const char *format, ...)\n+{\n+\tva_list ap;\n+\n+\tva_start(ap, format);\n+\tmsg_with_prefix(\"# \", format, ap);\n+\tva_end(ap);\n+}\n+\n+void test_plan(int count)\n+{\n+\tassert(!ctx.running);\n+\n+\tfflush(stderr);\n+\tprintf(\"1..%d\\n\", count);\n+\tfflush(stdout);\n+\tctx.lazy_plan = 0;\n+}\n+\n+int test_done(void)\n+{\n+\tassert(!ctx.running);\n+\n+\tif (ctx.lazy_plan)\n+\t\ttest_plan(ctx.count);\n+\n+\treturn ctx.failed;\n+}\n+\n+void test_skip(const char *format, ...)\n+{\n+\tva_list ap;\n+\n+\tassert(ctx.running);\n+\n+\tctx.result = RESULT_SKIP;\n+\tva_start(ap, format);\n+\tif (format)\n+\t\tmsg_with_prefix(\"# skipping test - \", format, ap);\n+\tva_end(ap);\n+}\n+\n+void test_skip_all(const char *format, ...)\n+{\n+\tva_list ap;\n+\tconst char *prefix;\n+\n+\tif (!ctx.count && ctx.lazy_plan) {\n+\t\t/* We have not printed a test plan yet */\n+\t\tprefix = \"1..0 # SKIP \";\n+\t\tctx.lazy_plan = 0;\n+\t} else {\n+\t\t/* We have already printed a test plan */\n+\t\tprefix = \"Bail out! # \";\n+\t\tctx.failed = 1;\n+\t}\n+\tctx.skip_all = 1;\n+\tctx.result = RESULT_SKIP;\n+\tva_start(ap, format);\n+\tmsg_with_prefix(prefix, format, ap);\n+\tva_end(ap);\n+}\n+\n+int test__run_begin(void)\n+{\n+\tassert(!ctx.running);\n+\n+\tctx.count++;\n+\tctx.result = RESULT_NONE;\n+\tctx.running = 1;\n+\n+\treturn ctx.skip_all;\n+}\n+\n+static void print_description(const char *format, va_list ap)\n+{\n+\tif (format) {\n+\t\tfputs(\" - \", stdout);\n+\t\tvprintf(format, ap);\n+\t}\n+}\n+\n+int test__run_end(int was_run UNUSED, const char *location, const char *format, ...)\n+{\n+\tva_list ap;\n+\n+\tassert(ctx.running);\n+\tassert(!ctx.todo);\n+\n+\tfflush(stderr);\n+\tva_start(ap, format);\n+\tif (!ctx.skip_all) {\n+\t\tswitch (ctx.result) {\n+\t\tcase RESULT_SUCCESS:\n+\t\t\tprintf(\"ok %d\", ctx.count);\n+\t\t\tprint_description(format, ap);\n+\t\t\tbreak;\n+\n+\t\tcase RESULT_FAILURE:\n+\t\t\tprintf(\"not ok %d\", ctx.count);\n+\t\t\tprint_description(format, ap);\n+\t\t\tbreak;\n+\n+\t\tcase RESULT_TODO:\n+\t\t\tprintf(\"not ok %d\", ctx.count);\n+\t\t\tprint_description(format, ap);\n+\t\t\tprintf(\" # TODO\");\n+\t\t\tbreak;\n+\n+\t\tcase RESULT_SKIP:\n+\t\t\tprintf(\"ok %d\", ctx.count);\n+\t\t\tprint_description(format, ap);\n+\t\t\tprintf(\" # SKIP\");\n+\t\t\tbreak;\n+\n+\t\tcase RESULT_NONE:\n+\t\t\ttest_msg(\"BUG: test has no checks at %s\", location);\n+\t\t\tprintf(\"not ok %d\", ctx.count);\n+\t\t\tprint_description(format, ap);\n+\t\t\tctx.result = RESULT_FAILURE;\n+\t\t\tbreak;\n+\t\t}\n+\t}\n+\tva_end(ap);\n+\tctx.running = 0;\n+\tif (ctx.skip_all)\n+\t\treturn 0;\n+\tputc('\\n', stdout);\n+\tfflush(stdout);\n+\tctx.failed |= ctx.result == RESULT_FAILURE;\n+\n+\treturn -(ctx.result == RESULT_FAILURE);\n+}\n+\n+static void test_fail(void)\n+{\n+\tassert(ctx.result != RESULT_SKIP);\n+\n+\tctx.result = RESULT_FAILURE;\n+}\n+\n+static void test_pass(void)\n+{\n+\tassert(ctx.result != RESULT_SKIP);\n+\n+\tif (ctx.result == RESULT_NONE)\n+\t\tctx.result = RESULT_SUCCESS;\n+}\n+\n+static void test_todo(void)\n+{\n+\tassert(ctx.result != RESULT_SKIP);\n+\n+\tif (ctx.result != RESULT_FAILURE)\n+\t\tctx.result = RESULT_TODO;\n+}\n+\n+int test_assert(const char *location, const char *check, int ok)\n+{\n+\tassert(ctx.running);\n+\n+\tif (ctx.result == RESULT_SKIP) {\n+\t\ttest_msg(\"skipping check '%s' at %s\", check, location);\n+\t\treturn 0;\n+\t} else if (!ctx.todo) {\n+\t\tif (ok) {\n+\t\t\ttest_pass();\n+\t\t} else {\n+\t\t\ttest_msg(\"check \\\"%s\\\" failed at %s\", check, location);\n+\t\t\ttest_fail();\n+\t\t}\n+\t}\n+\n+\treturn -!ok;\n+}\n+\n+void test__todo_begin(void)\n+{\n+\tassert(ctx.running);\n+\tassert(!ctx.todo);\n+\n+\tctx.todo = 1;\n+}\n+\n+int test__todo_end(const char *location, const char *check, int res)\n+{\n+\tassert(ctx.running);\n+\tassert(ctx.todo);\n+\n+\tctx.todo = 0;\n+\tif (ctx.result == RESULT_SKIP)\n+\t\treturn 0;\n+\tif (!res) {\n+\t\ttest_msg(\"todo check '%s' succeeded at %s\", check, location);\n+\t\ttest_fail();\n+\t} else {\n+\t\ttest_todo();\n+\t}\n+\n+\treturn -!res;\n+}\n+\n+int check_bool_loc(const char *loc, const char *check, int ok)\n+{\n+\treturn test_assert(loc, check, ok);\n+}\n+\n+union test__tmp test__tmp[2];\n+\n+int check_int_loc(const char *loc, const char *check, int ok,\n+\t\t intmax_t a, intmax_t b)\n+{\n+\tint ret = test_assert(loc, check, ok);\n+\n+\tif (ret) {\n+\t\ttest_msg(\" left: %\"PRIdMAX, a);\n+\t\ttest_msg(\" right: %\"PRIdMAX, b);\n+\t}\n+\n+\treturn ret;\n+}\n+\n+int check_uint_loc(const char *loc, const char *check, int ok,\n+\t\t uintmax_t a, uintmax_t b)\n+{\n+\tint ret = test_assert(loc, check, ok);\n+\n+\tif (ret) {\n+\t\ttest_msg(\" left: %\"PRIuMAX, a);\n+\t\ttest_msg(\" right: %\"PRIuMAX, b);\n+\t}\n+\n+\treturn ret;\n+}\n+\n+static void print_one_char(char ch, char quote)\n+{\n+\tif ((unsigned char)ch < 0x20u || ch == 0x7f) {\n+\t\t/* TODO: improve handling of \\a, \\b, \\f ... */\n+\t\tprintf(\"\\\\%03o\", (unsigned char)ch);\n+\t} else {\n+\t\tif (ch == '\\\\' || ch == quote)\n+\t\t\tputc('\\\\', stdout);\n+\t\tputc(ch, stdout);\n+\t}\n+}\n+\n+static void print_char(const char *prefix, char ch)\n+{\n+\tprintf(\"# %s: '\", prefix);\n+\tprint_one_char(ch, '\\'');\n+\tfputs(\"'\\n\", stdout);\n+}\n+\n+int check_char_loc(const char *loc, const char *check, int ok, char a, char b)\n+{\n+\tint ret = test_assert(loc, check, ok);\n+\n+\tif (ret) {\n+\t\tfflush(stderr);\n+\t\tprint_char(\" left\", a);\n+\t\tprint_char(\" right\", b);\n+\t\tfflush(stdout);\n+\t}\n+\n+\treturn ret;\n+}\n+\n+static void print_str(const char *prefix, const char *str)\n+{\n+\tprintf(\"# %s: \", prefix);\n+\tif (!str) {\n+\t\tfputs(\"NULL\\n\", stdout);\n+\t} else {\n+\t\tputc('\"', stdout);\n+\t\twhile (*str)\n+\t\t\tprint_one_char(*str++, '\"');\n+\t\tfputs(\"\\\"\\n\", stdout);\n+\t}\n+}\n+\n+int check_str_loc(const char *loc, const char *check,\n+\t\t const char *a, const char *b)\n+{\n+\tint ok = (!a && !b) || (a && b && !strcmp(a, b));\n+\tint ret = test_assert(loc, check, ok);\n+\n+\tif (ret) {\n+\t\tfflush(stderr);\n+\t\tprint_str(\" left\", a);\n+\t\tprint_str(\" right\", b);\n+\t\tfflush(stdout);\n+\t}\n+\n+\treturn ret;\n+}\ndiff --git a/t/unit-tests/test-lib.h b/t/unit-tests/test-lib.h\nnew file mode 100644\nindex 0000000000..720c97c6f8\n--- /dev/null\n+++ b/t/unit-tests/test-lib.h\n@@ -0,0 +1,143 @@\n+#ifndef TEST_LIB_H\n+#define TEST_LIB_H\n+\n+#include \"git-compat-util.h\"\n+\n+/*\n+ * Run a test function, returns 0 if the test succeeds, -1 if it\n+ * fails. If test_skip_all() has been called then the test will not be\n+ * run. The description for each test should be unique. For example:\n+ *\n+ * TEST(test_something(arg1, arg2), \"something %d %d\", arg1, arg2)\n+ */\n+#define TEST(t, ...)\t\t\t\t\t\\\n+\ttest__run_end(test__run_begin() ? 0 : (t, 1),\t\\\n+\t\t TEST_LOCATION(), __VA_ARGS__)\n+\n+/*\n+ * Print a test plan, should be called before any tests. If the number\n+ * of tests is not known in advance test_done() will automatically\n+ * print a plan at the end of the test program.\n+ */\n+void test_plan(int count);\n+\n+/*\n+ * test_done() must be called at the end of main(). It will print the\n+ * plan if plan() was not called at the beginning of the test program\n+ * and returns the exit code for the test program.\n+ */\n+int test_done(void);\n+\n+/* Skip the current test. */\n+__attribute__((format (printf, 1, 2)))\n+void test_skip(const char *format, ...);\n+\n+/* Skip all remaining tests. */\n+__attribute__((format (printf, 1, 2)))\n+void test_skip_all(const char *format, ...);\n+\n+/* Print a diagnostic message to stdout. */\n+__attribute__((format (printf, 1, 2)))\n+void test_msg(const char *format, ...);\n+\n+/*\n+ * Test checks are built around test_assert(). checks return 0 on\n+ * success, -1 on failure. If any check fails then the test will\n+ * fail. To create a custom check define a function that wraps\n+ * test_assert() and a macro to wrap that function. For example:\n+ *\n+ * static int check_oid_loc(const char *loc, const char *check,\n+ *\t\t\t struct object_id *a, struct object_id *b)\n+ * {\n+ *\t int res = test_assert(loc, check, oideq(a, b));\n+ *\n+ *\t if (res) {\n+ *\t\t test_msg(\" left: %s\", oid_to_hex(a);\n+ *\t\t test_msg(\" right: %s\", oid_to_hex(a);\n+ *\n+ *\t }\n+ *\t return res;\n+ * }\n+ *\n+ * #define check_oid(a, b) \\\n+ *\t check_oid_loc(TEST_LOCATION(), \"oideq(\"#a\", \"#b\")\", a, b)\n+ */\n+int test_assert(const char *location, const char *check, int ok);\n+\n+/* Helper macro to pass the location to checks */\n+#define TEST_LOCATION() TEST__MAKE_LOCATION(__LINE__)\n+\n+/* Check a boolean condition. */\n+#define check(x)\t\t\t\t\\\n+\tcheck_bool_loc(TEST_LOCATION(), #x, x)\n+int check_bool_loc(const char *loc, const char *check, int ok);\n+\n+/*\n+ * Compare two integers. Prints a message with the two values if the\n+ * comparison fails. NB this is not thread safe.\n+ */\n+#define check_int(a, op, b)\t\t\t\t\t\t\\\n+\t(test__tmp[0].i = (a), test__tmp[1].i = (b),\t\t\t\\\n+\t check_int_loc(TEST_LOCATION(), #a\" \"#op\" \"#b,\t\t\t\\\n+\t\t test__tmp[0].i op test__tmp[1].i, a, b))\n+int check_int_loc(const char *loc, const char *check, int ok,\n+\t\t intmax_t a, intmax_t b);\n+\n+/*\n+ * Compare two unsigned integers. Prints a message with the two values\n+ * if the comparison fails. NB this is not thread safe.\n+ */\n+#define check_uint(a, op, b)\t\t\t\t\t\t\\\n+\t(test__tmp[0].u = (a), test__tmp[1].u = (b),\t\t\t\\\n+\t check_uint_loc(TEST_LOCATION(), #a\" \"#op\" \"#b,\t\t\t\\\n+\t\t\ttest__tmp[0].u op test__tmp[1].u, a, b))\n+int check_uint_loc(const char *loc, const char *check, int ok,\n+\t\t uintmax_t a, uintmax_t b);\n+\n+/*\n+ * Compare two chars. Prints a message with the two values if the\n+ * comparison fails. NB this is not thread safe.\n+ */\n+#define check_char(a, op, b)\t\t\t\t\t\t\\\n+\t(test__tmp[0].c = (a), test__tmp[1].c = (b),\t\t\t\\\n+\t check_char_loc(TEST_LOCATION(), #a\" \"#op\" \"#b,\t\t\t\\\n+\t\t\ttest__tmp[0].c op test__tmp[1].c, a, b))\n+int check_char_loc(const char *loc, const char *check, int ok,\n+\t\t char a, char b);\n+\n+/* Check whether two strings are equal. */\n+#define check_str(a, b)\t\t\t\t\t\t\t\\\n+\tcheck_str_loc(TEST_LOCATION(), \"!strcmp(\"#a\", \"#b\")\", a, b)\n+int check_str_loc(const char *loc, const char *check,\n+\t\t const char *a, const char *b);\n+\n+/*\n+ * Wrap a check that is known to fail. If the check succeeds then the\n+ * test will fail. Returns 0 if the check fails, -1 if it\n+ * succeeds. For example:\n+ *\n+ * TEST_TODO(check(0));\n+ */\n+#define TEST_TODO(check) \\\n+\t(test__todo_begin(), test__todo_end(TEST_LOCATION(), #check, check))\n+\n+/* Private helpers */\n+\n+#define TEST__STR(x) #x\n+#define TEST__MAKE_LOCATION(line) __FILE__ \":\" TEST__STR(line)\n+\n+union test__tmp {\n+\tintmax_t i;\n+\tuintmax_t u;\n+\tchar c;\n+};\n+\n+extern union test__tmp test__tmp[2];\n+\n+int test__run_begin(void);\n+__attribute__((format (printf, 3, 4)))\n+int test__run_end(int, const char *, const char *, ...);\n+void test__todo_begin(void);\n+int test__todo_end(const char *, const char *, int);\n+\n+#endif /* TEST_LIB_H */\n-- \n2.41.0.694.ge786442a9b-goog\n\n"},{"id":"480734","messageId":"xmqqpm3mcyrc.fsf@gitster.g","threadId":"59937","inReplyTo":"ca284c575ece0aee7149641d5fb1977ccd7e7873.1692229626.git.steadmon@google.com","subject":"Re: [PATCH v6 2/3] unit tests: add TAP unit test framework","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2023-08-17T00:12:55Z","receivedAt":"2023-08-17T00:13:36Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Josh Steadmon writes:\n\n> diff --git a/Makefile b/Makefile\n> index e440728c24..4016da6e39 100644\n>\n> --- a/Makefile\n> +++ b/Makefile\n\nWith that blank line, I seem to be getting\n\n Applying: unit tests: add TAP unit test framework\n error: patch with only garbage at line 3\n Patch failed at 0002 unit tests: add TAP unit test framework\t\n\nAnd with that blank line removed, I seem to then get\n\n Applying: unit tests: add TAP unit test framework\n error: patch failed: Makefile:682\n error: Makefile: patch does not apply\n error: patch failed: t/Makefile:41\n error: t/Makefile: patch does not apply\n\nThis is on top of \"The fifth batch\", the commit your cover letter\nrefers to as the base of the series, so I am puzzled...\n\n"},{"id":"480735","messageId":"xmqqleeacxfj.fsf@gitster.g","threadId":"59937","inReplyTo":"xmqqpm3mcyrc.fsf@gitster.g","subject":"Re: [PATCH v6 2/3] unit tests: add TAP unit test framework","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2023-08-17T00:41:36Z","receivedAt":"2023-08-17T00:42:24Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Junio C Hamano writes:\n\n> Josh Steadmon writes:\n>\n>> diff --git a/Makefile b/Makefile\n>> index e440728c24..4016da6e39 100644\n>>\n>> --- a/Makefile\n>> +++ b/Makefile\n>\n> With that blank line, I seem to be getting\n>\n> Applying: unit tests: add TAP unit test framework\n> error: patch with only garbage at line 3\n> Patch failed at 0002 unit tests: add TAP unit test framework\t\n>\n> And with that blank line removed, I seem to then get\n>\n> Applying: unit tests: add TAP unit test framework\n> error: patch failed: Makefile:682\n> error: Makefile: patch does not apply\n> error: patch failed: t/Makefile:41\n> error: t/Makefile: patch does not apply\n>\n> This is on top of \"The fifth batch\", the commit your cover letter\n> refers to as the base of the series, so I am puzzled...\n\nWell, I suspected that 2/3 comes from\nhttps://lore.kernel.org/git/c902a166-98ce-afba-93f2-ea6027557176@gmail.com/\nwhich itself is whitespace damaged but has a reference to the\nunit-tests branch of https://github.com/phillipwood/git repository.\n\nBut it seems to be different in subtle ways.\n\nPlease send a set of patches that can be applied cleanly (especially\nwhen it is not an RFC series).\n\nThanks.\n"},{"id":"480745","messageId":"e4a91a8f-0157-4b75-b864-31e37741d9ac@gmail.com","threadId":"59937","inReplyTo":"ZNwCSdrbuG3K/iMP@google.com","subject":"Re: [PATCH v5] unit tests: Add a project plan document","fromName":"Phillip Wood","fromEmail":"phillip.wood123@gmail.com","sentAt":"2023-08-17T09:05:40Z","receivedAt":"2023-08-17T09:06:32Z","isPatch":true,"sender":{"key":"phillip.wood@dunelm.org.uk","avatar":null},"body":"Hi Josh\n\nOn 15/08/2023 23:55, Josh Steadmon wrote:\n> On 2023.08.14 14:29, Phillip Wood wrote:\n>> [...]\n\n>> I don't have a strong preference for which harness we use so long as it\n>> provides a way to (a) run tests that previously failed tests first and (b)\n>> run slow tests first. I do have a strong preference for using the same\n>> harness for both the unit tests and the integration tests so developers\n>> don't have to learn two different tools. Unless there is a problem with\n>> prove it would probably make sense just to keep using that as the project\n>> test harness.\n> \n> To be clear, it sounds like both of these can be done with `prove`\n> (using the various --state settings) without any further support from\n> our unit tests, right? \n\nYes\n\n> I see that we do have a \"failed\" target for\n> re-running integration tests, but that relies on some test-lib.sh\n> features that currently have no equivalent in the unit test framework.\n\nOoh, I didn't know about that, I think we could add something similar to \nthe framework if we wanted.\n\n> [...]\n>> It sounds like we're getting to the point where we have pinned down our\n>> requirements and the available alternatives well enough to make a decision.\n> \n> Yes, v6 will include your TAP implementation (I assume you are still OK\n> if I include your patch in this series?).\n\nYes that's fine. I'm about to go off the list for a couple of weeks, \nI'll take a proper look at v6 once I'm back.\n\nBest Wishes\n\nPhillip\n"},{"id":"480752","messageId":"ZN5oJM0B0DWohw64@google.com","threadId":"59937","inReplyTo":"xmqqleeacxfj.fsf@gitster.g","subject":"Re: [PATCH v6 2/3] unit tests: add TAP unit test framework","fromName":"Josh Steadmon","fromEmail":"steadmon@google.com","sentAt":"2023-08-17T18:34:12Z","receivedAt":"2023-08-17T18:35:23Z","isPatch":true,"sender":{"key":"steadmon@google.com","avatar":"https://avatars.githubusercontent.com/u/2654920?v=4"},"body":"On 2023.08.16 17:41, Junio C Hamano wrote:\n> Junio C Hamano writes:\n> \n> > Josh Steadmon writes:\n> >\n> >> diff --git a/Makefile b/Makefile\n> >> index e440728c24..4016da6e39 100644\n> >>\n> >> --- a/Makefile\n> >> +++ b/Makefile\n> >\n> > With that blank line, I seem to be getting\n> >\n> > Applying: unit tests: add TAP unit test framework\n> > error: patch with only garbage at line 3\n> > Patch failed at 0002 unit tests: add TAP unit test framework\t\n> >\n> > And with that blank line removed, I seem to then get\n> >\n> > Applying: unit tests: add TAP unit test framework\n> > error: patch failed: Makefile:682\n> > error: Makefile: patch does not apply\n> > error: patch failed: t/Makefile:41\n> > error: t/Makefile: patch does not apply\n> >\n> > This is on top of \"The fifth batch\", the commit your cover letter\n> > refers to as the base of the series, so I am puzzled...\n> \n> Well, I suspected that 2/3 comes from\n> https://lore.kernel.org/git/c902a166-98ce-afba-93f2-ea6027557176@gmail.com/\n> which itself is whitespace damaged but has a reference to the\n> unit-tests branch of https://github.com/phillipwood/git repository.\n> \n> But it seems to be different in subtle ways.\n> \n> Please send a set of patches that can be applied cleanly (especially\n> when it is not an RFC series).\n> \n> Thanks.\n\nSorry about the noise; I'm not sure how but the patch 2 commit message\ngot a partial diff pasted in. I've fixed this in v7 which I'll send\nshortly.\n"},{"id":"480753","messageId":"cover.1692297001.git.steadmon@google.com","threadId":"59937","inReplyTo":"0169ce6fb9ccafc089b74ae406db0d1a8ff8ac65.1688165272.git.steadmon@google.com","subject":"[PATCH v7 0/3] Add unit test framework and project plan","fromName":"Josh Steadmon","fromEmail":"steadmon@google.com","sentAt":"2023-08-17T18:37:20Z","receivedAt":"2023-08-17T18:38:39Z","isPatch":true,"sender":{"key":"steadmon@google.com","avatar":"https://avatars.githubusercontent.com/u/2654920?v=4"},"body":"In our current testing environment, we spend a significant amount of\neffort crafting end-to-end tests for error conditions that could easily\nbe captured by unit tests (or we simply forgo some hard-to-setup and\nrare error conditions). Unit tests additionally provide stability to the\ncodebase and can simplify debugging through isolation. Turning parts of\nGit into libraries[1] gives us the ability to run unit tests on the\nlibraries and to write unit tests in C. Writing unit tests in pure C,\nrather than with our current shell/test-tool helper setup, simplifies\ntest setup, simplifies passing data around (no shell-isms required), and\nreduces testing runtime by not spawning a separate process for every\ntest invocation.\n\nThis series begins with a project document covering our goals for adding\nunit tests and a discussion of alternative frameworks considered, as\nwell as the features used to evaluate them. A rendered preview of this\ndoc can be found at [2]. It also adds Phillip Wood's TAP implemenation\n(with some slightly re-worked Makefile rules) and a sample strbuf unit\ntest. Finally, we modify the configs for GitHub and Cirrus CI to run the\nunit tests. Sample runs showing successful CI runs can be found at [3],\n[4], and [5].\n\n[1] https://lore.kernel.org/git/CAJoAoZ=Cig_kLocxKGax31sU7Xe4==BGzC__Bg2_pr7krNq6MA@mail.gmail.com/\n[2] https://github.com/steadmon/git/blob/unit-tests-asciidoc/Documentation/technical/unit-tests.adoc\n[3] https://github.com/steadmon/git/actions/runs/5884659246/job/15959781385#step:4:1803\n[4] https://github.com/steadmon/git/actions/runs/5884659246/job/15959938401#step:5:186\n[5] https://cirrus-ci.com/task/6126304366428160 (unrelated tests failed,\n but note that t-strbuf ran successfully)\n\nIn addition to reviewing the patches in this series, reviewers can help\nthis series progress by chiming in on these remaining TODOs:\n- Figure out how to ensure tests run on additional OSes such as NonStop\n- Figure out if we should collect unit tests statistics similar to the\n \"counts\" files for shell tests\n- Decide if it's OK to wait on sharding unit tests across \"sliced\" CI\n instances\n- Provide guidelines for writing new unit tests\n\nChanges in v7:\n- Fix corrupt diff in patch #2, sorry for the noise.\n\nChanges in v6:\n- Officially recommend using Phillip Wood's TAP framework\n- Add an example strbuf unit test using the TAP framework as well as\n Makefile integration\n- Run unit tests in CI\n\nChanges in v5:\n- Add comparison point \"License\".\n- Discuss feature priorities\n- Drop frameworks:\n - Incompatible licenses: libtap, cmocka\n - Missing source: MyTAP\n - No TAP support: µnit, cmockery, cmockery2, Unity, minunit, CUnit\n- Drop comparison point \"Coverage reports\": this can generally be\n handled by tools such as `gcov` regardless of the framework used.\n- Drop comparison point \"Inline tests\": there didn't seem to be\n strong interest from reviewers for this feature.\n- Drop comparison point \"Scheduling / re-running\": this was not\n supported by any of the main contenders, and is generally better\n handled by the harness rather than framework.\n- Drop comparison point \"Lazy test planning\": this was supported by\n all frameworks that provide TAP output.\n\nChanges in v4:\n- Add link anchors for the framework comparison dimensions\n- Explain \"Partial\" results for each dimension\n- Use consistent dimension names in the section headers and comparison\n tables\n- Add \"Project KLOC\", \"Adoption\", and \"Inline tests\" dimensions\n- Fill in a few of the missing entries in the comparison table\n\nChanges in v3:\n- Expand the doc with discussion of desired features and a WIP\n comparison.\n- Drop all implementation patches until a framework is selected.\n- Link to v2: https://lore.kernel.org/r/20230517-unit-tests-v2-v2-0-21b5b60f4b32@google.com\n\n\nJosh Steadmon (2):\n unit tests: Add a project plan document\n ci: run unit tests in CI\n\nPhillip Wood (1):\n unit tests: add TAP unit test framework\n\n .cirrus.yml | 2 +-\n Documentation/Makefile | 1 +\n Documentation/technical/unit-tests.txt | 220 +++++++++++++++++\n Makefile | 24 +-\n ci/run-build-and-tests.sh | 2 +\n ci/run-test-slice.sh | 5 +\n t/Makefile | 15 +-\n t/t0080-unit-test-output.sh | 58 +++++\n t/unit-tests/.gitignore | 2 +\n t/unit-tests/t-basic.c | 95 +++++++\n t/unit-tests/t-strbuf.c | 75 ++++++\n t/unit-tests/test-lib.c | 329 +++++++++++++++++++++++++\n t/unit-tests/test-lib.h | 143 +++++++++++\n 13 files changed, 966 insertions(+), 5 deletions(-)\n create mode 100644 Documentation/technical/unit-tests.txt\n create mode 100755 t/t0080-unit-test-output.sh\n create mode 100644 t/unit-tests/.gitignore\n create mode 100644 t/unit-tests/t-basic.c\n create mode 100644 t/unit-tests/t-strbuf.c\n create mode 100644 t/unit-tests/test-lib.c\n create mode 100644 t/unit-tests/test-lib.h\n\nRange-diff against v6:\n-: ---------- > 1: 81c5148a12 unit tests: Add a project plan document\n1: ca284c575e ! 2: 3cc98d4045 unit tests: add TAP unit test framework\n @@ Commit message\n Signed-off-by: Phillip Wood \n Signed-off-by: Josh Steadmon \n \n - diff --git a/Makefile b/Makefile\n - index e440728c24..4016da6e39 100644\n -\n - --- a/Makefile\n - +++ b/Makefile\n - @@ -682,6 +682,8 @@ TEST_BUILTINS_OBJS =\n - TEST_OBJS =\n - TEST_PROGRAMS_NEED_X =\n - THIRD_PARTY_SOURCES =\n - +UNIT_TEST_PROGRAMS =\n - +UNIT_TEST_DIR = t/unit-tests\n -\n - # Having this variable in your environment would break pipelines because\n - # you cause \"cd\" to echo its destination to stdout. It can also take\n - @@ -1331,6 +1333,12 @@ THIRD_PARTY_SOURCES += compat/regex/%\n - THIRD_PARTY_SOURCES += sha1collisiondetection/%\n - THIRD_PARTY_SOURCES += sha1dc/%\n -\n - +UNIT_TEST_PROGRAMS += t-basic\n - +UNIT_TEST_PROGRAMS += t-strbuf\n - +UNIT_TEST_PROGS = $(patsubst %,$(UNIT_TEST_DIR)/%$X,$(UNIT_TEST_PROGRAMS))\n - +UNIT_TEST_OBJS = $(patsubst %,$(UNIT_TEST_DIR)/%.o,$(UNIT_TEST_PROGRAMS))\n - +UNIT_TEST_OBJS += $(UNIT_TEST_DIR)/test-lib.o\n - +\n - # xdiff and reftable libs may in turn depend on what is in libgit.a\n - GITLIBS = common-main.o $(LIB_FILE) $(XDIFF_LIB) $(REFTABLE_LIB) $(LIB_FILE)\n - EXTLIBS =\n - @@ -2672,6 +2680,7 @@ OBJECTS += $(TEST_OBJS)\n - OBJECTS += $(XDIFF_OBJS)\n - OBJECTS += $(FUZZ_OBJS)\n - OBJECTS += $(REFTABLE_OBJS) $(REFTABLE_TEST_OBJS)\n - +OBJECTS += $(UNIT_TEST_OBJS)\n -\n - ifndef NO_CURL\n - OBJECTS += http.o http-walker.o remote-curl.o\n - @@ -3167,7 +3176,7 @@ endif\n -\n - test_bindir_programs := $(patsubst %,bin-wrappers/%,$(BINDIR_PROGRAMS_NEED_X) $(BINDIR_PROGRAMS_NO_X) $(TEST_PROGRAMS_NEED_X))\n -\n - -all:: $(TEST_PROGRAMS) $(test_bindir_programs)\n - +all:: $(TEST_PROGRAMS) $(test_bindir_programs) $(UNIT_TEST_PROGS)\n -\n - bin-wrappers/%: wrap-for-bin.sh\n - $(call mkdir_p_parent_template)\n - @@ -3592,7 +3601,7 @@ endif\n -\n - artifacts-tar:: $(ALL_COMMANDS_TO_INSTALL) $(SCRIPT_LIB) $(OTHER_PROGRAMS) \\\n - GIT-BUILD-OPTIONS $(TEST_PROGRAMS) $(test_bindir_programs) \\\n - - $(MOFILES)\n - + $(UNIT_TEST_PROGS) $(MOFILES)\n - $(QUIET_SUBDIR0)templates $(QUIET_SUBDIR1) \\\n - SHELL_PATH='$(SHELL_PATH_SQ)' PERL_PATH='$(PERL_PATH_SQ)'\n - test -n \"$(ARTIFACTS_DIRECTORY)\"\n - @@ -3653,7 +3662,7 @@ clean: profile-clean coverage-clean cocciclean\n - $(RM) $(OBJECTS)\n - $(RM) $(LIB_FILE) $(XDIFF_LIB) $(REFTABLE_LIB) $(REFTABLE_TEST_LIB)\n - $(RM) $(ALL_PROGRAMS) $(SCRIPT_LIB) $(BUILT_INS) $(OTHER_PROGRAMS)\n - - $(RM) $(TEST_PROGRAMS)\n - + $(RM) $(TEST_PROGRAMS) $(UNIT_TEST_PROGS)\n - $(RM) $(FUZZ_PROGRAMS)\n - $(RM) $(SP_OBJ)\n - $(RM) $(HCC)\n - @@ -3831,3 +3840,12 @@ $(FUZZ_PROGRAMS): all\n - $(XDIFF_OBJS) $(EXTLIBS) git.o $@.o $(LIB_FUZZING_ENGINE) -o $@\n -\n - fuzz-all: $(FUZZ_PROGRAMS)\n - +\n - +$(UNIT_TEST_PROGS): $(UNIT_TEST_DIR)/%$X: $(UNIT_TEST_DIR)/%.o $(UNIT_TEST_DIR)/test-lib.o $(GITLIBS) GIT-LDFLAGS\n - + $(QUIET_LINK)$(CC) $(ALL_CFLAGS) -o $@ $(ALL_LDFLAGS) \\\n - + $(filter %.o,$^) $(filter %.a,$^) $(LIBS)\n - +\n - +.PHONY: build-unit-tests unit-tests\n - +build-unit-tests: $(UNIT_TEST_PROGS)\n - +unit-tests: $(UNIT_TEST_PROGS)\n - + $(MAKE) -C t/ unit-tests\n - diff --git a/t/Makefile b/t/Makefile\n - index 3e00cdd801..92864cdf28 100644\n - --- a/t/Makefile\n - +++ b/t/Makefile\n - @@ -41,6 +41,7 @@ TPERF = $(sort $(wildcard perf/p[0-9][0-9][0-9][0-9]-*.sh))\n - TINTEROP = $(sort $(wildcard interop/i[0-9][0-9][0-9][0-9]-*.sh))\n - CHAINLINTTESTS = $(sort $(patsubst chainlint/%.test,%,$(wildcard chainlint/*.test)))\n - CHAINLINT = '$(PERL_PATH_SQ)' chainlint.pl\n - +UNIT_TESTS = $(sort $(filter-out %.h %.c %.o unit-tests/t-basic%,$(wildcard unit-tests/*)))\n -\n - # `test-chainlint` (which is a dependency of `test-lint`, `test` and `prove`)\n - # checks all tests in all scripts via a single invocation, so tell individual\n - @@ -65,6 +66,13 @@ prove: pre-clean check-chainlint $(TEST_LINT)\n - $(T):\n - @echo \"*** $@ ***\"; '$(TEST_SHELL_PATH_SQ)' $@ $(GIT_TEST_OPTS)\n -\n - +$(UNIT_TESTS):\n - + @echo \"*** $@ ***\"; $@\n - +\n - +.PHONY: unit-tests\n - +unit-tests:\n - + @echo \"*** prove - unit tests ***\"; $(PROVE) $(GIT_PROVE_OPTS) $(UNIT_TESTS)\n - +\n - pre-clean:\n - $(RM) -r '$(TEST_RESULTS_DIRECTORY_SQ)'\n -\n - @@ -149,4 +157,4 @@ perf:\n - $(MAKE) -C perf/ all\n -\n - .PHONY: pre-clean $(T) aggregate-results clean valgrind perf \\\n - - check-chainlint clean-chainlint test-chainlint\n - + check-chainlint clean-chainlint test-chainlint $(UNIT_TESTS)\n - diff --git a/t/t0080-unit-test-output.sh b/t/t0080-unit-test-output.sh\n - new file mode 100755\n - index 0000000000..c60e402260\n - --- /dev/null\n - +++ b/t/t0080-unit-test-output.sh\n - @@ -0,0 +1,58 @@\n - +#!/bin/sh\n - +\n - +test_description='Test the output of the unit test framework'\n - +\n - +. ./test-lib.sh\n - +\n - +test_expect_success 'TAP output from unit tests' '\n - + cat >expect <<-EOF &&\n - + ok 1 - passing test\n - + ok 2 - passing test and assertion return 0\n - + # check \"1 == 2\" failed at t/unit-tests/t-basic.c:68\n - + # left: 1\n - + # right: 2\n - + not ok 3 - failing test\n - + ok 4 - failing test and assertion return -1\n - + not ok 5 - passing TEST_TODO() # TODO\n - + ok 6 - passing TEST_TODO() returns 0\n - + # todo check ${SQ}check(x)${SQ} succeeded at t/unit-tests/t-basic.c:17\n - + not ok 7 - failing TEST_TODO()\n - + ok 8 - failing TEST_TODO() returns -1\n - + # check \"0\" failed at t/unit-tests/t-basic.c:22\n - + # skipping test - missing prerequisite\n - + # skipping check ${SQ}1${SQ} at t/unit-tests/t-basic.c:24\n - + ok 9 - test_skip() # SKIP\n - + ok 10 - skipped test returns 0\n - + # skipping test - missing prerequisite\n - + ok 11 - test_skip() inside TEST_TODO() # SKIP\n - + ok 12 - test_skip() inside TEST_TODO() returns 0\n - + # check \"0\" failed at t/unit-tests/t-basic.c:40\n - + not ok 13 - TEST_TODO() after failing check\n - + ok 14 - TEST_TODO() after failing check returns -1\n - + # check \"0\" failed at t/unit-tests/t-basic.c:48\n - + not ok 15 - failing check after TEST_TODO()\n - + ok 16 - failing check after TEST_TODO() returns -1\n - + # check \"!strcmp(\"\\thello\\\\\\\\\", \"there\\\"\\n\")\" failed at t/unit-tests/t-basic.c:53\n - + # left: \"\\011hello\\\\\\\\\"\n - + # right: \"there\\\"\\012\"\n - + # check \"!strcmp(\"NULL\", NULL)\" failed at t/unit-tests/t-basic.c:54\n - + # left: \"NULL\"\n - + # right: NULL\n - + # check \"${SQ}a${SQ} == ${SQ}\\n${SQ}\" failed at t/unit-tests/t-basic.c:55\n - + # left: ${SQ}a${SQ}\n - + # right: ${SQ}\\012${SQ}\n - + # check \"${SQ}\\\\\\\\${SQ} == ${SQ}\\\\${SQ}${SQ}\" failed at t/unit-tests/t-basic.c:56\n - + # left: ${SQ}\\\\\\\\${SQ}\n - + # right: ${SQ}\\\\${SQ}${SQ}\n - + not ok 17 - messages from failing string and char comparison\n - + # BUG: test has no checks at t/unit-tests/t-basic.c:83\n - + not ok 18 - test with no checks\n - + ok 19 - test with no checks returns -1\n - + 1..19\n - + EOF\n - +\n - + ! \"$GIT_BUILD_DIR\"/t/unit-tests/t-basic >actual &&\n - + test_cmp expect actual\n - +'\n - +\n - +test_done\n - diff --git a/t/unit-tests/.gitignore b/t/unit-tests/.gitignore\n - new file mode 100644\n - index 0000000000..e292d58348\n - --- /dev/null\n - +++ b/t/unit-tests/.gitignore\n - @@ -0,0 +1,2 @@\n - +/t-basic\n - +/t-strbuf\n - diff --git a/t/unit-tests/t-basic.c b/t/unit-tests/t-basic.c\n - new file mode 100644\n - index 0000000000..ab0b7682c4\n - --- /dev/null\n - +++ b/t/unit-tests/t-basic.c\n - @@ -0,0 +1,87 @@\n - +#include \"test-lib.h\"\n - +\n - +/* Used to store the return value of check_int(). */\n - +static int check_res;\n - +\n - +/* Used to store the return value of TEST(). */\n - +static int test_res;\n - +\n - +static void t_res(int expect)\n - +{\n - + check_int(check_res, ==, expect);\n - + check_int(test_res, ==, expect);\n - +}\n - +\n - +static void t_todo(int x)\n - +{\n - + check_res = TEST_TODO(check(x));\n - +}\n - +\n - +static void t_skip(void)\n - +{\n - + check(0);\n - + test_skip(\"missing prerequisite\");\n - + check(1);\n - +}\n - +\n - +static int do_skip(void)\n - +{\n - + test_skip(\"missing prerequisite\");\n - + return 0;\n - +}\n - +\n - +static void t_skip_todo(void)\n - +{\n - + check_res = TEST_TODO(do_skip());\n - +}\n - +\n - +static void t_todo_after_fail(void)\n - +{\n - + check(0);\n - + TEST_TODO(check(0));\n - +}\n - +\n - +static void t_fail_after_todo(void)\n - +{\n - + check(1);\n - + TEST_TODO(check(0));\n - + check(0);\n - +}\n - +\n - +static void t_messages(void)\n - +{\n - + check_str(\"\\thello\\\\\", \"there\\\"\\n\");\n - + check_str(\"NULL\", NULL);\n - + check_char('a', ==, '\\n');\n - + check_char('\\\\', ==, '\\'');\n - +}\n - +\n - +static void t_empty(void)\n - +{\n - + ; /* empty */\n - +}\n - +\n - +int cmd_main(int argc, const char **argv)\n - +{\n - + test_res = TEST(check_res = check_int(1, ==, 1), \"passing test\");\n - + TEST(t_res(0), \"passing test and assertion return 0\");\n - + test_res = TEST(check_res = check_int(1, ==, 2), \"failing test\");\n - + TEST(t_res(-1), \"failing test and assertion return -1\");\n - + test_res = TEST(t_todo(0), \"passing TEST_TODO()\");\n - + TEST(t_res(0), \"passing TEST_TODO() returns 0\");\n - + test_res = TEST(t_todo(1), \"failing TEST_TODO()\");\n - + TEST(t_res(-1), \"failing TEST_TODO() returns -1\");\n - + test_res = TEST(t_skip(), \"test_skip()\");\n - + TEST(check_int(test_res, ==, 0), \"skipped test returns 0\");\n - + test_res = TEST(t_skip_todo(), \"test_skip() inside TEST_TODO()\");\n - + TEST(t_res(0), \"test_skip() inside TEST_TODO() returns 0\");\n - + test_res = TEST(t_todo_after_fail(), \"TEST_TODO() after failing check\");\n - + TEST(check_int(test_res, ==, -1), \"TEST_TODO() after failing check returns -1\");\n - + test_res = TEST(t_fail_after_todo(), \"failing check after TEST_TODO()\");\n - + TEST(check_int(test_res, ==, -1), \"failing check after TEST_TODO() returns -1\");\n - + TEST(t_messages(), \"messages from failing string and char comparison\");\n - + test_res = TEST(t_empty(), \"test with no checks\");\n - + TEST(check_int(test_res, ==, -1), \"test with no checks returns -1\");\n - +\n - + return test_done();\n - +}\n - diff --git a/t/unit-tests/t-strbuf.c b/t/unit-tests/t-strbuf.c\n - new file mode 100644\n - index 0000000000..561611e242\n - --- /dev/null\n - +++ b/t/unit-tests/t-strbuf.c\n - @@ -0,0 +1,75 @@\n - +#include \"test-lib.h\"\n - +#include \"strbuf.h\"\n - +\n - +/* wrapper that supplies tests with an initialized strbuf */\n - +static void setup(void (*f)(struct strbuf*, void*), void *data)\n - +{\n - + struct strbuf buf = STRBUF_INIT;\n - +\n - + f(&buf, data);\n - + strbuf_release(&buf);\n - + check_uint(buf.len, ==, 0);\n - + check_uint(buf.alloc, ==, 0);\n - + check(buf.buf == strbuf_slopbuf);\n - + check_char(buf.buf[0], ==, '\\0');\n - +}\n - +\n - +static void t_static_init(void)\n - +{\n - + struct strbuf buf = STRBUF_INIT;\n - +\n - + check_uint(buf.len, ==, 0);\n - + check_uint(buf.alloc, ==, 0);\n - + if (check(buf.buf == strbuf_slopbuf))\n - + return; /* avoid de-referencing buf.buf */\n - + check_char(buf.buf[0], ==, '\\0');\n - +}\n - +\n - +static void t_dynamic_init(void)\n - +{\n - + struct strbuf buf;\n - +\n - + strbuf_init(&buf, 1024);\n - + check_uint(buf.len, ==, 0);\n - + check_uint(buf.alloc, >=, 1024);\n - + check_char(buf.buf[0], ==, '\\0');\n - + strbuf_release(&buf);\n - +}\n - +\n - +static void t_addch(struct strbuf *buf, void *data)\n - +{\n - + const char *p_ch = data;\n - + const char ch = *p_ch;\n - +\n - + strbuf_addch(buf, ch);\n - + if (check_uint(buf->len, ==, 1) ||\n - + check_uint(buf->alloc, >, 1))\n - + return; /* avoid de-referencing buf->buf */\n - + check_char(buf->buf[0], ==, ch);\n - + check_char(buf->buf[1], ==, '\\0');\n - +}\n - +\n - +static void t_addstr(struct strbuf *buf, void *data)\n - +{\n - + const char *text = data;\n - + size_t len = strlen(text);\n - +\n - + strbuf_addstr(buf, text);\n - + if (check_uint(buf->len, ==, len) ||\n - + check_uint(buf->alloc, >, len) ||\n - + check_char(buf->buf[len], ==, '\\0'))\n - + return;\n - + check_str(buf->buf, text);\n - +}\n - +\n - +int cmd_main(int argc, const char **argv)\n - +{\n - + if (TEST(t_static_init(), \"static initialization works\"))\n - + test_skip_all(\"STRBUF_INIT is broken\");\n - + TEST(t_dynamic_init(), \"dynamic initialization works\");\n - + TEST(setup(t_addch, \"a\"), \"strbuf_addch adds char\");\n - + TEST(setup(t_addch, \"\"), \"strbuf_addch adds NUL char\");\n - + TEST(setup(t_addstr, \"hello there\"), \"strbuf_addstr adds string\");\n - +\n - + return test_done();\n - +}\n - diff --git a/t/unit-tests/test-lib.c b/t/unit-tests/test-lib.c\n - new file mode 100644\n - index 0000000000..70030d587f\n - --- /dev/null\n - +++ b/t/unit-tests/test-lib.c\n - @@ -0,0 +1,329 @@\n - +#include \"test-lib.h\"\n - +\n - +enum result {\n - + RESULT_NONE,\n - + RESULT_FAILURE,\n - + RESULT_SKIP,\n - + RESULT_SUCCESS,\n - + RESULT_TODO\n - +};\n - +\n - +static struct {\n - + enum result result;\n - + int count;\n - + unsigned failed :1;\n - + unsigned lazy_plan :1;\n - + unsigned running :1;\n - + unsigned skip_all :1;\n - + unsigned todo :1;\n - +} ctx = {\n - + .lazy_plan = 1,\n - + .result = RESULT_NONE,\n - +};\n - +\n - +static void msg_with_prefix(const char *prefix, const char *format, va_list ap)\n - +{\n - + fflush(stderr);\n - + if (prefix)\n - + fprintf(stdout, \"%s\", prefix);\n - + vprintf(format, ap); /* TODO: handle newlines */\n - + putc('\\n', stdout);\n - + fflush(stdout);\n - +}\n - +\n - +void test_msg(const char *format, ...)\n - +{\n - + va_list ap;\n - +\n - + va_start(ap, format);\n - + msg_with_prefix(\"# \", format, ap);\n - + va_end(ap);\n - +}\n - +\n - +void test_plan(int count)\n - +{\n - + assert(!ctx.running);\n - +\n - + fflush(stderr);\n - + printf(\"1..%d\\n\", count);\n - + fflush(stdout);\n - + ctx.lazy_plan = 0;\n - +}\n - +\n - +int test_done(void)\n - +{\n - + assert(!ctx.running);\n - +\n - + if (ctx.lazy_plan)\n - + test_plan(ctx.count);\n - +\n - + return ctx.failed;\n - +}\n - +\n - +void test_skip(const char *format, ...)\n - +{\n - + va_list ap;\n - +\n - + assert(ctx.running);\n - +\n - + ctx.result = RESULT_SKIP;\n - + va_start(ap, format);\n - + if (format)\n - + msg_with_prefix(\"# skipping test - \", format, ap);\n - + va_end(ap);\n - +}\n - +\n - +void test_skip_all(const char *format, ...)\n - +{\n - + va_list ap;\n - + const char *prefix;\n - +\n - + if (!ctx.count && ctx.lazy_plan) {\n - + /* We have not printed a test plan yet */\n - + prefix = \"1..0 # SKIP \";\n - + ctx.lazy_plan = 0;\n - + } else {\n - + /* We have already printed a test plan */\n - + prefix = \"Bail out! # \";\n - + ctx.failed = 1;\n - + }\n - + ctx.skip_all = 1;\n - + ctx.result = RESULT_SKIP;\n - + va_start(ap, format);\n - + msg_with_prefix(prefix, format, ap);\n - + va_end(ap);\n - +}\n - +\n - +int test__run_begin(void)\n - +{\n - + assert(!ctx.running);\n - +\n - + ctx.count++;\n - + ctx.result = RESULT_NONE;\n - + ctx.running = 1;\n - +\n - + return ctx.skip_all;\n - +}\n - +\n - +static void print_description(const char *format, va_list ap)\n - +{\n - + if (format) {\n - + fputs(\" - \", stdout);\n - + vprintf(format, ap);\n - + }\n - +}\n - +\n - +int test__run_end(int was_run UNUSED, const char *location, const char *format, ...)\n - +{\n - + va_list ap;\n - +\n - + assert(ctx.running);\n - + assert(!ctx.todo);\n - +\n - + fflush(stderr);\n - + va_start(ap, format);\n - + if (!ctx.skip_all) {\n - + switch (ctx.result) {\n - + case RESULT_SUCCESS:\n - + printf(\"ok %d\", ctx.count);\n - + print_description(format, ap);\n - + break;\n - +\n - + case RESULT_FAILURE:\n - + printf(\"not ok %d\", ctx.count);\n - + print_description(format, ap);\n - + break;\n - +\n - + case RESULT_TODO:\n - + printf(\"not ok %d\", ctx.count);\n - + print_description(format, ap);\n - + printf(\" # TODO\");\n - + break;\n - +\n - + case RESULT_SKIP:\n - + printf(\"ok %d\", ctx.count);\n - + print_description(format, ap);\n - + printf(\" # SKIP\");\n - + break;\n - +\n - + case RESULT_NONE:\n - + test_msg(\"BUG: test has no checks at %s\", location);\n - + printf(\"not ok %d\", ctx.count);\n - + print_description(format, ap);\n - + ctx.result = RESULT_FAILURE;\n - + break;\n - + }\n - + }\n - + va_end(ap);\n - + ctx.running = 0;\n - + if (ctx.skip_all)\n - + return 0;\n - + putc('\\n', stdout);\n - + fflush(stdout);\n - + ctx.failed |= ctx.result == RESULT_FAILURE;\n - +\n - + return -(ctx.result == RESULT_FAILURE);\n - +}\n - +\n - +static void test_fail(void)\n - +{\n - + assert(ctx.result != RESULT_SKIP);\n - +\n - + ctx.result = RESULT_FAILURE;\n - +}\n - +\n - +static void test_pass(void)\n - +{\n - + assert(ctx.result != RESULT_SKIP);\n - +\n - + if (ctx.result == RESULT_NONE)\n - + ctx.result = RESULT_SUCCESS;\n - +}\n - +\n - +static void test_todo(void)\n - +{\n - + assert(ctx.result != RESULT_SKIP);\n - +\n - + if (ctx.result != RESULT_FAILURE)\n - + ctx.result = RESULT_TODO;\n - +}\n - +\n - +int test_assert(const char *location, const char *check, int ok)\n - +{\n - + assert(ctx.running);\n - +\n - + if (ctx.result == RESULT_SKIP) {\n - + test_msg(\"skipping check '%s' at %s\", check, location);\n - + return 0;\n - + } else if (!ctx.todo) {\n - + if (ok) {\n - + test_pass();\n - + } else {\n - + test_msg(\"check \\\"%s\\\" failed at %s\", check, location);\n - + test_fail();\n - + }\n - + }\n - +\n - + return -!ok;\n - +}\n - +\n - +void test__todo_begin(void)\n - +{\n - + assert(ctx.running);\n - + assert(!ctx.todo);\n - +\n - + ctx.todo = 1;\n - +}\n - +\n - +int test__todo_end(const char *location, const char *check, int res)\n - +{\n - + assert(ctx.running);\n - + assert(ctx.todo);\n - +\n - + ctx.todo = 0;\n - + if (ctx.result == RESULT_SKIP)\n - + return 0;\n - + if (!res) {\n - + test_msg(\"todo check '%s' succeeded at %s\", check, location);\n - + test_fail();\n - + } else {\n - + test_todo();\n - + }\n - +\n - + return -!res;\n - +}\n - +\n - +int check_bool_loc(const char *loc, const char *check, int ok)\n - +{\n - + return test_assert(loc, check, ok);\n - +}\n - +\n - +union test__tmp test__tmp[2];\n - +\n - +int check_int_loc(const char *loc, const char *check, int ok,\n - + intmax_t a, intmax_t b)\n - +{\n - + int ret = test_assert(loc, check, ok);\n - +\n - + if (ret) {\n - + test_msg(\" left: %\"PRIdMAX, a);\n - + test_msg(\" right: %\"PRIdMAX, b);\n - + }\n - +\n - + return ret;\n - +}\n - +\n - +int check_uint_loc(const char *loc, const char *check, int ok,\n - + uintmax_t a, uintmax_t b)\n - +{\n - + int ret = test_assert(loc, check, ok);\n - +\n - + if (ret) {\n - + test_msg(\" left: %\"PRIuMAX, a);\n - + test_msg(\" right: %\"PRIuMAX, b);\n - + }\n - +\n - + return ret;\n - +}\n - +\n - +static void print_one_char(char ch, char quote)\n - +{\n - + if ((unsigned char)ch < 0x20u || ch == 0x7f) {\n - + /* TODO: improve handling of \\a, \\b, \\f ... */\n - + printf(\"\\\\%03o\", (unsigned char)ch);\n - + } else {\n - + if (ch == '\\\\' || ch == quote)\n - + putc('\\\\', stdout);\n - + putc(ch, stdout);\n - + }\n - +}\n - +\n - +static void print_char(const char *prefix, char ch)\n - +{\n - + printf(\"# %s: '\", prefix);\n - + print_one_char(ch, '\\'');\n - + fputs(\"'\\n\", stdout);\n - +}\n - +\n - +int check_char_loc(const char *loc, const char *check, int ok, char a, char b)\n - +{\n - + int ret = test_assert(loc, check, ok);\n - +\n - + if (ret) {\n - + fflush(stderr);\n - + print_char(\" left\", a);\n - + print_char(\" right\", b);\n - + fflush(stdout);\n - + }\n - +\n - + return ret;\n - +}\n - +\n - +static void print_str(const char *prefix, const char *str)\n - +{\n - + printf(\"# %s: \", prefix);\n - + if (!str) {\n - + fputs(\"NULL\\n\", stdout);\n - + } else {\n - + putc('\"', stdout);\n - + while (*str)\n - + print_one_char(*str++, '\"');\n - + fputs(\"\\\"\\n\", stdout);\n - + }\n - +}\n - +\n - +int check_str_loc(const char *loc, const char *check,\n - + const char *a, const char *b)\n - +{\n - + int ok = (!a && !b) || (a && b && !strcmp(a, b));\n - + int ret = test_assert(loc, check, ok);\n - +\n - + if (ret) {\n - + fflush(stderr);\n - + print_str(\" left\", a);\n - + print_str(\" right\", b);\n - + fflush(stdout);\n - + }\n - +\n - + return ret;\n - +}\n - diff --git a/t/unit-tests/test-lib.h b/t/unit-tests/test-lib.h\n - new file mode 100644\n - index 0000000000..720c97c6f8\n - --- /dev/null\n - +++ b/t/unit-tests/test-lib.h\n - @@ -0,0 +1,143 @@\n - +#ifndef TEST_LIB_H\n - +#define TEST_LIB_H\n - +\n - +#include \"git-compat-util.h\"\n - +\n - +/*\n - + * Run a test function, returns 0 if the test succeeds, -1 if it\n - + * fails. If test_skip_all() has been called then the test will not be\n - + * run. The description for each test should be unique. For example:\n - + *\n - + * TEST(test_something(arg1, arg2), \"something %d %d\", arg1, arg2)\n - + */\n - +#define TEST(t, ...) \\\n - + test__run_end(test__run_begin() ? 0 : (t, 1), \\\n - + TEST_LOCATION(), __VA_ARGS__)\n - +\n - +/*\n - + * Print a test plan, should be called before any tests. If the number\n - + * of tests is not known in advance test_done() will automatically\n - + * print a plan at the end of the test program.\n - + */\n - +void test_plan(int count);\n - +\n - +/*\n - + * test_done() must be called at the end of main(). It will print the\n - + * plan if plan() was not called at the beginning of the test program\n - + * and returns the exit code for the test program.\n - + */\n - +int test_done(void);\n - +\n - +/* Skip the current test. */\n - +__attribute__((format (printf, 1, 2)))\n - +void test_skip(const char *format, ...);\n - +\n - +/* Skip all remaining tests. */\n - +__attribute__((format (printf, 1, 2)))\n - +void test_skip_all(const char *format, ...);\n - +\n - +/* Print a diagnostic message to stdout. */\n - +__attribute__((format (printf, 1, 2)))\n - +void test_msg(const char *format, ...);\n - +\n - +/*\n - + * Test checks are built around test_assert(). checks return 0 on\n - + * success, -1 on failure. If any check fails then the test will\n - + * fail. To create a custom check define a function that wraps\n - + * test_assert() and a macro to wrap that function. For example:\n - + *\n - + * static int check_oid_loc(const char *loc, const char *check,\n - + * struct object_id *a, struct object_id *b)\n - + * {\n - + * int res = test_assert(loc, check, oideq(a, b));\n - + *\n - + * if (res) {\n - + * test_msg(\" left: %s\", oid_to_hex(a);\n - + * test_msg(\" right: %s\", oid_to_hex(a);\n - + *\n - + * }\n - + * return res;\n - + * }\n - + *\n - + * #define check_oid(a, b) \\\n - + * check_oid_loc(TEST_LOCATION(), \"oideq(\"#a\", \"#b\")\", a, b)\n - + */\n - +int test_assert(const char *location, const char *check, int ok);\n - +\n - +/* Helper macro to pass the location to checks */\n - +#define TEST_LOCATION() TEST__MAKE_LOCATION(__LINE__)\n - +\n - +/* Check a boolean condition. */\n - +#define check(x) \\\n - + check_bool_loc(TEST_LOCATION(), #x, x)\n - +int check_bool_loc(const char *loc, const char *check, int ok);\n - +\n - +/*\n - + * Compare two integers. Prints a message with the two values if the\n - + * comparison fails. NB this is not thread safe.\n - + */\n - +#define check_int(a, op, b) \\\n - + (test__tmp[0].i = (a), test__tmp[1].i = (b), \\\n - + check_int_loc(TEST_LOCATION(), #a\" \"#op\" \"#b, \\\n - + test__tmp[0].i op test__tmp[1].i, a, b))\n - +int check_int_loc(const char *loc, const char *check, int ok,\n - + intmax_t a, intmax_t b);\n - +\n - +/*\n - + * Compare two unsigned integers. Prints a message with the two values\n - + * if the comparison fails. NB this is not thread safe.\n - + */\n - +#define check_uint(a, op, b) \\\n - + (test__tmp[0].u = (a), test__tmp[1].u = (b), \\\n - + check_uint_loc(TEST_LOCATION(), #a\" \"#op\" \"#b, \\\n - + test__tmp[0].u op test__tmp[1].u, a, b))\n - +int check_uint_loc(const char *loc, const char *check, int ok,\n - + uintmax_t a, uintmax_t b);\n - +\n - +/*\n - + * Compare two chars. Prints a message with the two values if the\n - + * comparison fails. NB this is not thread safe.\n - + */\n - +#define check_char(a, op, b) \\\n - + (test__tmp[0].c = (a), test__tmp[1].c = (b), \\\n - + check_char_loc(TEST_LOCATION(), #a\" \"#op\" \"#b, \\\n - + test__tmp[0].c op test__tmp[1].c, a, b))\n - +int check_char_loc(const char *loc, const char *check, int ok,\n - + char a, char b);\n - +\n - +/* Check whether two strings are equal. */\n - +#define check_str(a, b) \\\n - + check_str_loc(TEST_LOCATION(), \"!strcmp(\"#a\", \"#b\")\", a, b)\n - +int check_str_loc(const char *loc, const char *check,\n - + const char *a, const char *b);\n - +\n - +/*\n - + * Wrap a check that is known to fail. If the check succeeds then the\n - + * test will fail. Returns 0 if the check fails, -1 if it\n - + * succeeds. For example:\n - + *\n - + * TEST_TODO(check(0));\n - + */\n - +#define TEST_TODO(check) \\\n - + (test__todo_begin(), test__todo_end(TEST_LOCATION(), #check, check))\n - +\n - +/* Private helpers */\n - +\n - +#define TEST__STR(x) #x\n - +#define TEST__MAKE_LOCATION(line) __FILE__ \":\" TEST__STR(line)\n - +\n - +union test__tmp {\n - + intmax_t i;\n - + uintmax_t u;\n - + char c;\n - +};\n - +\n - +extern union test__tmp test__tmp[2];\n - +\n - +int test__run_begin(void);\n - +__attribute__((format (printf, 3, 4)))\n - +int test__run_end(int, const char *, const char *, ...);\n - +void test__todo_begin(void);\n - +int test__todo_end(const char *, const char *, int);\n - +\n - +#endif /* TEST_LIB_H */\n -\n ## Makefile ##\n @@ Makefile: TEST_BUILTINS_OBJS =\n TEST_OBJS =\n2: ea33518d00 = 3: abf4dc41ac ci: run unit tests in CI\n\nbase-commit: a9e066fa63149291a55f383cfa113d8bdbdaa6b3\n-- \n2.42.0.rc1.204.g551eb34607-goog\n\n"},{"id":"480754","messageId":"abf4dc41ac73d6bd13e338e0031cc12e0eb3e271.1692297001.git.steadmon@google.com","threadId":"59937","inReplyTo":"cover.1692297001.git.steadmon@google.com","subject":"[PATCH v7 3/3] ci: run unit tests in CI","fromName":"Josh Steadmon","fromEmail":"steadmon@google.com","sentAt":"2023-08-17T18:37:23Z","receivedAt":"2023-08-17T18:38:39Z","isPatch":true,"sender":{"key":"steadmon@google.com","avatar":"https://avatars.githubusercontent.com/u/2654920?v=4"},"body":"Run unit tests in both Cirrus and GitHub CI. For sharded CI instances\n(currently just Windows on GitHub), run only on the first shard. This is\nOK while we have only a single unit test executable, but we may wish to\ndistribute tests more evenly when we add new unit tests in the future.\n\nWe may also want to add more status output in our unit test framework,\nso that we can do similar post-processing as in\nci/lib.sh:handle_failed_tests().\n\nSigned-off-by: Josh Steadmon \n---\n .cirrus.yml | 2 +-\n ci/run-build-and-tests.sh | 2 ++\n ci/run-test-slice.sh | 5 +++++\n t/Makefile | 2 +-\n 4 files changed, 9 insertions(+), 2 deletions(-)\n\ndiff --git a/.cirrus.yml b/.cirrus.yml\nindex 4860bebd32..b6280692d2 100644\n--- a/.cirrus.yml\n+++ b/.cirrus.yml\n@@ -19,4 +19,4 @@ freebsd_12_task:\n build_script:\n - su git -c gmake\n test_script:\n- - su git -c 'gmake test'\n+ - su git -c 'gmake DEFAULT_UNIT_TEST_TARGET=unit-tests-prove test unit-tests'\ndiff --git a/ci/run-build-and-tests.sh b/ci/run-build-and-tests.sh\nindex 2528f25e31..7a1466b868 100755\n--- a/ci/run-build-and-tests.sh\n+++ b/ci/run-build-and-tests.sh\n@@ -50,6 +50,8 @@ if test -n \"$run_tests\"\n then\n \tgroup \"Run tests\" make test ||\n \thandle_failed_tests\n+\tgroup \"Run unit tests\" \\\n+\t\tmake DEFAULT_UNIT_TEST_TARGET=unit-tests-prove unit-tests\n fi\n check_unignored_build_artifacts\n \ndiff --git a/ci/run-test-slice.sh b/ci/run-test-slice.sh\nindex a3c67956a8..ae8094382f 100755\n--- a/ci/run-test-slice.sh\n+++ b/ci/run-test-slice.sh\n@@ -15,4 +15,9 @@ group \"Run tests\" make --quiet -C t T=\"$(cd t &&\n \ttr '\\n' ' ')\" ||\n handle_failed_tests\n \n+# We only have one unit test at the moment, so run it in the first slice\n+if [ \"$1\" == \"0\" ] ; then\n+\tgroup \"Run unit tests\" make --quiet -C t unit-tests-prove\n+fi\n+\n check_unignored_build_artifacts\ndiff --git a/t/Makefile b/t/Makefile\nindex 2db8b3adb1..095334bfde 100644\n--- a/t/Makefile\n+++ b/t/Makefile\n@@ -42,7 +42,7 @@ TPERF = $(sort $(wildcard perf/p[0-9][0-9][0-9][0-9]-*.sh))\n TINTEROP = $(sort $(wildcard interop/i[0-9][0-9][0-9][0-9]-*.sh))\n CHAINLINTTESTS = $(sort $(patsubst chainlint/%.test,%,$(wildcard chainlint/*.test)))\n CHAINLINT = '$(PERL_PATH_SQ)' chainlint.pl\n-UNIT_TESTS = $(sort $(filter-out %.h %.c %.o unit-tests/t-basic%,$(wildcard unit-tests/*)))\n+UNIT_TESTS = $(sort $(filter-out %.h %.c %.o unit-tests/t-basic%,$(wildcard unit-tests/t-*)))\n \n # `test-chainlint` (which is a dependency of `test-lint`, `test` and `prove`)\n # checks all tests in all scripts via a single invocation, so tell individual\n-- \n2.42.0.rc1.204.g551eb34607-goog\n\n"},{"id":"480755","messageId":"81c5148a1267b8f9ce432a950340f0fa16b4d773.1692297001.git.steadmon@google.com","threadId":"59937","inReplyTo":"cover.1692297001.git.steadmon@google.com","subject":"[PATCH v7 1/3] unit tests: Add a project plan document","fromName":"Josh Steadmon","fromEmail":"steadmon@google.com","sentAt":"2023-08-17T18:37:21Z","receivedAt":"2023-08-17T18:38:39Z","isPatch":true,"sender":{"key":"steadmon@google.com","avatar":"https://avatars.githubusercontent.com/u/2654920?v=4"},"body":"In our current testing environment, we spend a significant amount of\neffort crafting end-to-end tests for error conditions that could easily\nbe captured by unit tests (or we simply forgo some hard-to-setup and\nrare error conditions). Describe what we hope to accomplish by\nimplementing unit tests, and explain some open questions and milestones.\nDiscuss desired features for test frameworks/harnesses, and provide a\npreliminary comparison of several different frameworks.\n\nCo-authored-by: Calvin Wan \nSigned-off-by: Calvin Wan \nSigned-off-by: Josh Steadmon \n---\n Documentation/Makefile | 1 +\n Documentation/technical/unit-tests.txt | 220 +++++++++++++++++++++++++\n 2 files changed, 221 insertions(+)\n create mode 100644 Documentation/technical/unit-tests.txt\n\ndiff --git a/Documentation/Makefile b/Documentation/Makefile\nindex b629176d7d..3f2383a12c 100644\n--- a/Documentation/Makefile\n+++ b/Documentation/Makefile\n@@ -122,6 +122,7 @@ TECH_DOCS += technical/scalar\n TECH_DOCS += technical/send-pack-pipeline\n TECH_DOCS += technical/shallow\n TECH_DOCS += technical/trivial-merge\n+TECH_DOCS += technical/unit-tests\n SP_ARTICLES += $(TECH_DOCS)\n SP_ARTICLES += technical/api-index\n \ndiff --git a/Documentation/technical/unit-tests.txt b/Documentation/technical/unit-tests.txt\nnew file mode 100644\nindex 0000000000..b7a89cc838\n--- /dev/null\n+++ b/Documentation/technical/unit-tests.txt\n@@ -0,0 +1,220 @@\n+= Unit Testing\n+\n+In our current testing environment, we spend a significant amount of effort\n+crafting end-to-end tests for error conditions that could easily be captured by\n+unit tests (or we simply forgo some hard-to-setup and rare error conditions).\n+Unit tests additionally provide stability to the codebase and can simplify\n+debugging through isolation. Writing unit tests in pure C, rather than with our\n+current shell/test-tool helper setup, simplifies test setup, simplifies passing\n+data around (no shell-isms required), and reduces testing runtime by not\n+spawning a separate process for every test invocation.\n+\n+We believe that a large body of unit tests, living alongside the existing test\n+suite, will improve code quality for the Git project.\n+\n+== Definitions\n+\n+For the purposes of this document, we'll use *test framework* to refer to\n+projects that support writing test cases and running tests within the context\n+of a single executable. *Test harness* will refer to projects that manage\n+running multiple executables (each of which may contain multiple test cases) and\n+aggregating their results.\n+\n+In reality, these terms are not strictly defined, and many of the projects\n+discussed below contain features from both categories.\n+\n+For now, we will evaluate projects solely on their framework features. Since we\n+are relying on having TAP output (see below), we can assume that any framework\n+can be made to work with a harness that we can choose later.\n+\n+\n+== Choosing a framework\n+\n+We believe the best option is to implement a custom TAP framework for the Git\n+project. We use a version of the framework originally proposed in\n+https://lore.kernel.org/git/c902a166-98ce-afba-93f2-ea6027557176@gmail.com/[1].\n+\n+\n+== Choosing a test harness\n+\n+During upstream discussion, it was occasionally noted that `prove` provides many\n+convenient features, such as scheduling slower tests first, or re-running\n+previously failed tests.\n+\n+While we already support the use of `prove` as a test harness for the shell\n+tests, it is not strictly required. The t/Makefile allows running shell tests\n+directly (though with interleaved output if parallelism is enabled). Git\n+developers who wish to use `prove` as a more advanced harness can do so by\n+setting DEFAULT_TEST_TARGET=prove in their config.mak.\n+\n+We will follow a similar approach for unit tests: by default the test\n+executables will be run directly from the t/Makefile, but `prove` can be\n+configured with DEFAULT_UNIT_TEST_TARGET=prove.\n+\n+\n+== Framework selection\n+\n+There are a variety of features we can use to rank the candidate frameworks, and\n+those features have different priorities:\n+\n+* Critical features: we probably won't consider a framework without these\n+** Can we legally / easily use the project?\n+*** <>\n+*** <>\n+*** <>\n+*** <>\n+** Does the project support our bare-minimum needs?\n+*** <>\n+*** <>\n+*** <>\n+* Nice-to-have features:\n+** <>\n+** <>\n+** <>\n+* Tie-breaker stats\n+** <>\n+** <>\n+\n+[[license]]\n+=== License\n+\n+We must be able to legally use the framework in connection with Git. As Git is\n+licensed only under GPLv2, we must eliminate any LGPLv3, GPLv3, or Apache 2.0\n+projects.\n+\n+[[vendorable-or-ubiquitous]]\n+=== Vendorable or ubiquitous\n+\n+We want to avoid forcing Git developers to install new tools just to run unit\n+tests. Any prospective frameworks and harnesses must either be vendorable\n+(meaning, we can copy their source directly into Git's repository), or so\n+ubiquitous that it is reasonable to expect that most developers will have the\n+tools installed already.\n+\n+[[maintainable-extensible]]\n+=== Maintainable / extensible\n+\n+It is unlikely that any pre-existing project perfectly fits our needs, so any\n+project we select will need to be actively maintained and open to accepting\n+changes. Alternatively, assuming we are vendoring the source into our repo, it\n+must be simple enough that Git developers can feel comfortable making changes as\n+needed to our version.\n+\n+In the comparison table below, \"True\" means that the framework seems to have\n+active developers, that it is simple enough that Git developers can make changes\n+to it, and that the project seems open to accepting external contributions (or\n+that it is vendorable). \"Partial\" means that at least one of the above\n+conditions holds.\n+\n+[[major-platform-support]]\n+=== Major platform support\n+\n+At a bare minimum, unit-testing must work on Linux, MacOS, and Windows.\n+\n+In the comparison table below, \"True\" means that it works on all three major\n+platforms with no issues. \"Partial\" means that there may be annoyances on one or\n+more platforms, but it is still usable in principle.\n+\n+[[tap-support]]\n+=== TAP support\n+\n+The https://testanything.org/[Test Anything Protocol] is a text-based interface\n+that allows tests to communicate with a test harness. It is already used by\n+Git's integration test suite. Supporting TAP output is a mandatory feature for\n+any prospective test framework.\n+\n+In the comparison table below, \"True\" means this is natively supported.\n+\"Partial\" means TAP output must be generated by post-processing the native\n+output.\n+\n+Frameworks that do not have at least Partial support will not be evaluated\n+further.\n+\n+[[diagnostic-output]]\n+=== Diagnostic output\n+\n+When a test case fails, the framework must generate enough diagnostic output to\n+help developers find the appropriate test case in source code in order to debug\n+the failure.\n+\n+[[runtime-skippable-tests]]\n+=== Runtime-skippable tests\n+\n+Test authors may wish to skip certain test cases based on runtime circumstances,\n+so the framework should support this.\n+\n+[[parallel-execution]]\n+=== Parallel execution\n+\n+Ideally, we will build up a significant collection of unit test cases, most\n+likely split across multiple executables. It will be necessary to run these\n+tests in parallel to enable fast develop-test-debug cycles.\n+\n+In the comparison table below, \"True\" means that individual test cases within a\n+single test executable can be run in parallel. We assume that executable-level\n+parallelism can be handled by the test harness.\n+\n+[[mock-support]]\n+=== Mock support\n+\n+Unit test authors may wish to test code that interacts with objects that may be\n+inconvenient to handle in a test (e.g. interacting with a network service).\n+Mocking allows test authors to provide a fake implementation of these objects\n+for more convenient tests.\n+\n+[[signal-error-handling]]\n+=== Signal & error handling\n+\n+The test framework should fail gracefully when test cases are themselves buggy\n+or when they are interrupted by signals during runtime.\n+\n+[[project-kloc]]\n+=== Project KLOC\n+\n+The size of the project, in thousands of lines of code as measured by\n+https://dwheeler.com/sloccount/[sloccount] (rounded up to the next multiple of\n+1,000). As a tie-breaker, we probably prefer a project with fewer LOC.\n+\n+[[adoption]]\n+=== Adoption\n+\n+As a tie-breaker, we prefer a more widely-used project. We use the number of\n+GitHub / GitLab stars to estimate this.\n+\n+\n+=== Comparison\n+\n+[format=\"csv\",options=\"header\",width=\"33%\"]\n+|=====\n+Framework,\"<>\",\"<>\",\"<>\",\"<>\",\"<>\",\"<>\",\"<>\",\"<>\",\"<>\",\"<>\",\"<>\",\"<>\"\n+https://lore.kernel.org/git/c902a166-98ce-afba-93f2-ea6027557176@gmail.com/[Custom Git impl.],[lime-background]#GPL v2#,[lime-background]#True#,[lime-background]#True#,[lime-background]#True#,[lime-background]#True#,[lime-background]#True#,[lime-background]#True#,[red-background]#False#,[red-background]#False#,[red-background]#False#,1,0\n+https://github.com/silentbicycle/greatest[Greatest],[lime-background]#ISC#,[lime-background]#True#,[yellow-background]#Partial#,[lime-background]#True#,[yellow-background]#Partial#,[lime-background]#True#,[lime-background]#True#,[red-background]#False#,[red-background]#False#,[red-background]#False#,3,1400\n+https://github.com/Snaipe/Criterion[Criterion],[lime-background]#MIT#,[red-background]#False#,[yellow-background]#Partial#,[lime-background]#True#,[lime-background]#True#,[lime-background]#True#,[lime-background]#True#,[lime-background]#True#,[red-background]#False#,[lime-background]#True#,19,1800\n+https://github.com/rra/c-tap-harness/[C TAP],[lime-background]#Expat#,[lime-background]#True#,[yellow-background]#Partial#,[yellow-background]#Partial#,[lime-background]#True#,[red-background]#False#,[lime-background]#True#,[red-background]#False#,[red-background]#False#,[red-background]#False#,4,33\n+https://libcheck.github.io/check/[Check],[lime-background]#LGPL v2.1#,[red-background]#False#,[yellow-background]#Partial#,[lime-background]#True#,[lime-background]#True#,[lime-background]#True#,[red-background]#False#,[red-background]#False#,[red-background]#False#,[lime-background]#True#,17,973\n+|=====\n+\n+=== Additional framework candidates\n+\n+Several suggested frameworks have been eliminated from consideration:\n+\n+* Incompatible licenses:\n+** https://github.com/zorgnax/libtap[libtap] (LGPL v3)\n+** https://cmocka.org/[cmocka] (Apache 2.0)\n+* Missing source: https://www.kindahl.net/mytap/doc/index.html[MyTap]\n+* No TAP support:\n+** https://nemequ.github.io/munit/[µnit]\n+** https://github.com/google/cmockery[cmockery]\n+** https://github.com/lpabon/cmockery2[cmockery2]\n+** https://github.com/ThrowTheSwitch/Unity[Unity]\n+** https://github.com/siu/minunit[minunit]\n+** https://cunit.sourceforge.net/[CUnit]\n+\n+\n+== Milestones\n+\n+* Add useful tests of library-like code\n+* Integrate with\n+ https://lore.kernel.org/git/20230502211454.1673000-1-calvinwan@google.com/[stdlib\n+ work]\n+* Run alongside regular `make test` target\n-- \n2.42.0.rc1.204.g551eb34607-goog\n\n"},{"id":"480756","messageId":"3cc98d4045eeda6e8cc24914802edc16d367fba0.1692297001.git.steadmon@google.com","threadId":"59937","inReplyTo":"cover.1692297001.git.steadmon@google.com","subject":"[PATCH v7 2/3] unit tests: add TAP unit test framework","fromName":"Josh Steadmon","fromEmail":"steadmon@google.com","sentAt":"2023-08-17T18:37:22Z","receivedAt":"2023-08-17T18:38:40Z","isPatch":true,"sender":{"key":"steadmon@google.com","avatar":"https://avatars.githubusercontent.com/u/2654920?v=4"},"body":"From: Phillip Wood \n\nThis patch contains an implementation for writing unit tests with TAP\noutput. Each test is a function that contains one or more checks. The\ntest is run with the TEST() macro and if any of the checks fail then the\ntest will fail. A complete program that tests STRBUF_INIT would look\nlike\n\n #include \"test-lib.h\"\n #include \"strbuf.h\"\n\n static void t_static_init(void)\n {\n struct strbuf buf = STRBUF_INIT;\n\n check_uint(buf.len, ==, 0);\n check_uint(buf.alloc, ==, 0);\n if (check(buf.buf == strbuf_slopbuf))\n\t\t return; /* avoid SIGSEV */\n check_char(buf.buf[0], ==, '\\0');\n }\n\n int main(void)\n {\n TEST(t_static_init(), \"static initialization works);\n\n return test_done();\n }\n\nThe output of this program would be\n\n ok 1 - static initialization works\n 1..1\n\nIf any of the checks in a test fail then they print a diagnostic message\nto aid debugging and the test will be reported as failing. For example a\nfailing integer check would look like\n\n # check \"x >= 3\" failed at my-test.c:102\n # left: 2\n # right: 3\n not ok 1 - x is greater than or equal to three\n\nThere are a number of check functions implemented so far. check() checks\na boolean condition, check_int(), check_uint() and check_char() take two\nvalues to compare and a comparison operator. check_str() will check if\ntwo strings are equal. Custom checks are simple to implement as shown in\nthe comments above test_assert() in test-lib.h.\n\nTests can be skipped with test_skip() which can be supplied with a\nreason for skipping which it will print. Tests can print diagnostic\nmessages with test_msg(). Checks that are known to fail can be wrapped\nin TEST_TODO().\n\nThere are a couple of example test programs included in this\npatch. t-basic.c implements some self-tests and demonstrates the\ndiagnostic output for failing test. The output of this program is\nchecked by t0080-unit-test-output.sh. t-strbuf.c shows some example\nunit tests for strbuf.c\n\nThe unit tests can be built with \"make unit-tests\" (this works but the\nMakefile changes need some further work). Once they have been built they\ncan be run manually (e.g t/unit-tests/t-strbuf) or with prove.\n\nSigned-off-by: Phillip Wood \nSigned-off-by: Josh Steadmon \n---\n Makefile | 24 ++-\n t/Makefile | 15 +-\n t/t0080-unit-test-output.sh | 58 +++++++\n t/unit-tests/.gitignore | 2 +\n t/unit-tests/t-basic.c | 95 +++++++++++\n t/unit-tests/t-strbuf.c | 75 ++++++++\n t/unit-tests/test-lib.c | 329 ++++++++++++++++++++++++++++++++++++\n t/unit-tests/test-lib.h | 143 ++++++++++++++++\n 8 files changed, 737 insertions(+), 4 deletions(-)\n create mode 100755 t/t0080-unit-test-output.sh\n create mode 100644 t/unit-tests/.gitignore\n create mode 100644 t/unit-tests/t-basic.c\n create mode 100644 t/unit-tests/t-strbuf.c\n create mode 100644 t/unit-tests/test-lib.c\n create mode 100644 t/unit-tests/test-lib.h\n\ndiff --git a/Makefile b/Makefile\nindex e440728c24..4016da6e39 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -682,6 +682,8 @@ TEST_BUILTINS_OBJS =\n TEST_OBJS =\n TEST_PROGRAMS_NEED_X =\n THIRD_PARTY_SOURCES =\n+UNIT_TEST_PROGRAMS =\n+UNIT_TEST_DIR = t/unit-tests\n \n # Having this variable in your environment would break pipelines because\n # you cause \"cd\" to echo its destination to stdout. It can also take\n@@ -1331,6 +1333,12 @@ THIRD_PARTY_SOURCES += compat/regex/%\n THIRD_PARTY_SOURCES += sha1collisiondetection/%\n THIRD_PARTY_SOURCES += sha1dc/%\n \n+UNIT_TEST_PROGRAMS += t-basic\n+UNIT_TEST_PROGRAMS += t-strbuf\n+UNIT_TEST_PROGS = $(patsubst %,$(UNIT_TEST_DIR)/%$X,$(UNIT_TEST_PROGRAMS))\n+UNIT_TEST_OBJS = $(patsubst %,$(UNIT_TEST_DIR)/%.o,$(UNIT_TEST_PROGRAMS))\n+UNIT_TEST_OBJS += $(UNIT_TEST_DIR)/test-lib.o\n+\n # xdiff and reftable libs may in turn depend on what is in libgit.a\n GITLIBS = common-main.o $(LIB_FILE) $(XDIFF_LIB) $(REFTABLE_LIB) $(LIB_FILE)\n EXTLIBS =\n@@ -2672,6 +2680,7 @@ OBJECTS += $(TEST_OBJS)\n OBJECTS += $(XDIFF_OBJS)\n OBJECTS += $(FUZZ_OBJS)\n OBJECTS += $(REFTABLE_OBJS) $(REFTABLE_TEST_OBJS)\n+OBJECTS += $(UNIT_TEST_OBJS)\n \n ifndef NO_CURL\n \tOBJECTS += http.o http-walker.o remote-curl.o\n@@ -3167,7 +3176,7 @@ endif\n \n test_bindir_programs := $(patsubst %,bin-wrappers/%,$(BINDIR_PROGRAMS_NEED_X) $(BINDIR_PROGRAMS_NO_X) $(TEST_PROGRAMS_NEED_X))\n \n-all:: $(TEST_PROGRAMS) $(test_bindir_programs)\n+all:: $(TEST_PROGRAMS) $(test_bindir_programs) $(UNIT_TEST_PROGS)\n \n bin-wrappers/%: wrap-for-bin.sh\n \t$(call mkdir_p_parent_template)\n@@ -3592,7 +3601,7 @@ endif\n \n artifacts-tar:: $(ALL_COMMANDS_TO_INSTALL) $(SCRIPT_LIB) $(OTHER_PROGRAMS) \\\n \t\tGIT-BUILD-OPTIONS $(TEST_PROGRAMS) $(test_bindir_programs) \\\n-\t\t$(MOFILES)\n+\t\t$(UNIT_TEST_PROGS) $(MOFILES)\n \t$(QUIET_SUBDIR0)templates $(QUIET_SUBDIR1) \\\n \t\tSHELL_PATH='$(SHELL_PATH_SQ)' PERL_PATH='$(PERL_PATH_SQ)'\n \ttest -n \"$(ARTIFACTS_DIRECTORY)\"\n@@ -3653,7 +3662,7 @@ clean: profile-clean coverage-clean cocciclean\n \t$(RM) $(OBJECTS)\n \t$(RM) $(LIB_FILE) $(XDIFF_LIB) $(REFTABLE_LIB) $(REFTABLE_TEST_LIB)\n \t$(RM) $(ALL_PROGRAMS) $(SCRIPT_LIB) $(BUILT_INS) $(OTHER_PROGRAMS)\n-\t$(RM) $(TEST_PROGRAMS)\n+\t$(RM) $(TEST_PROGRAMS) $(UNIT_TEST_PROGS)\n \t$(RM) $(FUZZ_PROGRAMS)\n \t$(RM) $(SP_OBJ)\n \t$(RM) $(HCC)\n@@ -3831,3 +3840,12 @@ $(FUZZ_PROGRAMS): all\n \t\t$(XDIFF_OBJS) $(EXTLIBS) git.o $@.o $(LIB_FUZZING_ENGINE) -o $@\n \n fuzz-all: $(FUZZ_PROGRAMS)\n+\n+$(UNIT_TEST_PROGS): $(UNIT_TEST_DIR)/%$X: $(UNIT_TEST_DIR)/%.o $(UNIT_TEST_DIR)/test-lib.o $(GITLIBS) GIT-LDFLAGS\n+\t$(QUIET_LINK)$(CC) $(ALL_CFLAGS) -o $@ $(ALL_LDFLAGS) \\\n+\t\t$(filter %.o,$^) $(filter %.a,$^) $(LIBS)\n+\n+.PHONY: build-unit-tests unit-tests\n+build-unit-tests: $(UNIT_TEST_PROGS)\n+unit-tests: $(UNIT_TEST_PROGS)\n+\t$(MAKE) -C t/ unit-tests\ndiff --git a/t/Makefile b/t/Makefile\nindex 3e00cdd801..2db8b3adb1 100644\n--- a/t/Makefile\n+++ b/t/Makefile\n@@ -17,6 +17,7 @@ TAR ?= $(TAR)\n RM ?= rm -f\n PROVE ?= prove\n DEFAULT_TEST_TARGET ?= test\n+DEFAULT_UNIT_TEST_TARGET ?= unit-tests-raw\n TEST_LINT ?= test-lint\n \n ifdef TEST_OUTPUT_DIRECTORY\n@@ -41,6 +42,7 @@ TPERF = $(sort $(wildcard perf/p[0-9][0-9][0-9][0-9]-*.sh))\n TINTEROP = $(sort $(wildcard interop/i[0-9][0-9][0-9][0-9]-*.sh))\n CHAINLINTTESTS = $(sort $(patsubst chainlint/%.test,%,$(wildcard chainlint/*.test)))\n CHAINLINT = '$(PERL_PATH_SQ)' chainlint.pl\n+UNIT_TESTS = $(sort $(filter-out %.h %.c %.o unit-tests/t-basic%,$(wildcard unit-tests/*)))\n \n # `test-chainlint` (which is a dependency of `test-lint`, `test` and `prove`)\n # checks all tests in all scripts via a single invocation, so tell individual\n@@ -65,6 +67,17 @@ prove: pre-clean check-chainlint $(TEST_LINT)\n $(T):\n \t@echo \"*** $@ ***\"; '$(TEST_SHELL_PATH_SQ)' $@ $(GIT_TEST_OPTS)\n \n+$(UNIT_TESTS):\n+\t@echo \"*** $@ ***\"; $@\n+\n+.PHONY: unit-tests unit-tests-raw unit-tests-prove\n+unit-tests: $(DEFAULT_UNIT_TEST_TARGET)\n+\n+unit-tests-raw: $(UNIT_TESTS)\n+\n+unit-tests-prove:\n+\t@echo \"*** prove - unit tests ***\"; $(PROVE) $(GIT_PROVE_OPTS) $(UNIT_TESTS)\n+\n pre-clean:\n \t$(RM) -r '$(TEST_RESULTS_DIRECTORY_SQ)'\n \n@@ -149,4 +162,4 @@ perf:\n \t$(MAKE) -C perf/ all\n \n .PHONY: pre-clean $(T) aggregate-results clean valgrind perf \\\n-\tcheck-chainlint clean-chainlint test-chainlint\n+\tcheck-chainlint clean-chainlint test-chainlint $(UNIT_TESTS)\ndiff --git a/t/t0080-unit-test-output.sh b/t/t0080-unit-test-output.sh\nnew file mode 100755\nindex 0000000000..e0fc07d1e5\n--- /dev/null\n+++ b/t/t0080-unit-test-output.sh\n@@ -0,0 +1,58 @@\n+#!/bin/sh\n+\n+test_description='Test the output of the unit test framework'\n+\n+. ./test-lib.sh\n+\n+test_expect_success 'TAP output from unit tests' '\n+\tcat >expect <<-EOF &&\n+\tok 1 - passing test\n+\tok 2 - passing test and assertion return 0\n+\t# check \"1 == 2\" failed at t/unit-tests/t-basic.c:76\n+\t# left: 1\n+\t# right: 2\n+\tnot ok 3 - failing test\n+\tok 4 - failing test and assertion return -1\n+\tnot ok 5 - passing TEST_TODO() # TODO\n+\tok 6 - passing TEST_TODO() returns 0\n+\t# todo check ${SQ}check(x)${SQ} succeeded at t/unit-tests/t-basic.c:25\n+\tnot ok 7 - failing TEST_TODO()\n+\tok 8 - failing TEST_TODO() returns -1\n+\t# check \"0\" failed at t/unit-tests/t-basic.c:30\n+\t# skipping test - missing prerequisite\n+\t# skipping check ${SQ}1${SQ} at t/unit-tests/t-basic.c:32\n+\tok 9 - test_skip() # SKIP\n+\tok 10 - skipped test returns 0\n+\t# skipping test - missing prerequisite\n+\tok 11 - test_skip() inside TEST_TODO() # SKIP\n+\tok 12 - test_skip() inside TEST_TODO() returns 0\n+\t# check \"0\" failed at t/unit-tests/t-basic.c:48\n+\tnot ok 13 - TEST_TODO() after failing check\n+\tok 14 - TEST_TODO() after failing check returns -1\n+\t# check \"0\" failed at t/unit-tests/t-basic.c:56\n+\tnot ok 15 - failing check after TEST_TODO()\n+\tok 16 - failing check after TEST_TODO() returns -1\n+\t# check \"!strcmp(\"\\thello\\\\\\\\\", \"there\\\"\\n\")\" failed at t/unit-tests/t-basic.c:61\n+\t# left: \"\\011hello\\\\\\\\\"\n+\t# right: \"there\\\"\\012\"\n+\t# check \"!strcmp(\"NULL\", NULL)\" failed at t/unit-tests/t-basic.c:62\n+\t# left: \"NULL\"\n+\t# right: NULL\n+\t# check \"${SQ}a${SQ} == ${SQ}\\n${SQ}\" failed at t/unit-tests/t-basic.c:63\n+\t# left: ${SQ}a${SQ}\n+\t# right: ${SQ}\\012${SQ}\n+\t# check \"${SQ}\\\\\\\\${SQ} == ${SQ}\\\\${SQ}${SQ}\" failed at t/unit-tests/t-basic.c:64\n+\t# left: ${SQ}\\\\\\\\${SQ}\n+\t# right: ${SQ}\\\\${SQ}${SQ}\n+\tnot ok 17 - messages from failing string and char comparison\n+\t# BUG: test has no checks at t/unit-tests/t-basic.c:91\n+\tnot ok 18 - test with no checks\n+\tok 19 - test with no checks returns -1\n+\t1..19\n+\tEOF\n+\n+\t! \"$GIT_BUILD_DIR\"/t/unit-tests/t-basic >actual &&\n+\ttest_cmp expect actual\n+'\n+\n+test_done\ndiff --git a/t/unit-tests/.gitignore b/t/unit-tests/.gitignore\nnew file mode 100644\nindex 0000000000..e292d58348\n--- /dev/null\n+++ b/t/unit-tests/.gitignore\n@@ -0,0 +1,2 @@\n+/t-basic\n+/t-strbuf\ndiff --git a/t/unit-tests/t-basic.c b/t/unit-tests/t-basic.c\nnew file mode 100644\nindex 0000000000..d20f444fab\n--- /dev/null\n+++ b/t/unit-tests/t-basic.c\n@@ -0,0 +1,95 @@\n+#include \"test-lib.h\"\n+\n+/*\n+ * The purpose of this \"unit test\" is to verify a few invariants of the unit\n+ * test framework itself, as well as to provide examples of output from actually\n+ * failing tests. As such, it is intended that this test fails, and thus it\n+ * should not be run as part of `make unit-tests`. Instead, we verify it behaves\n+ * as expected in the integration test t0080-unit-test-output.sh\n+ */\n+\n+/* Used to store the return value of check_int(). */\n+static int check_res;\n+\n+/* Used to store the return value of TEST(). */\n+static int test_res;\n+\n+static void t_res(int expect)\n+{\n+\tcheck_int(check_res, ==, expect);\n+\tcheck_int(test_res, ==, expect);\n+}\n+\n+static void t_todo(int x)\n+{\n+\tcheck_res = TEST_TODO(check(x));\n+}\n+\n+static void t_skip(void)\n+{\n+\tcheck(0);\n+\ttest_skip(\"missing prerequisite\");\n+\tcheck(1);\n+}\n+\n+static int do_skip(void)\n+{\n+\ttest_skip(\"missing prerequisite\");\n+\treturn 0;\n+}\n+\n+static void t_skip_todo(void)\n+{\n+\tcheck_res = TEST_TODO(do_skip());\n+}\n+\n+static void t_todo_after_fail(void)\n+{\n+\tcheck(0);\n+\tTEST_TODO(check(0));\n+}\n+\n+static void t_fail_after_todo(void)\n+{\n+\tcheck(1);\n+\tTEST_TODO(check(0));\n+\tcheck(0);\n+}\n+\n+static void t_messages(void)\n+{\n+\tcheck_str(\"\\thello\\\\\", \"there\\\"\\n\");\n+\tcheck_str(\"NULL\", NULL);\n+\tcheck_char('a', ==, '\\n');\n+\tcheck_char('\\\\', ==, '\\'');\n+}\n+\n+static void t_empty(void)\n+{\n+\t; /* empty */\n+}\n+\n+int cmd_main(int argc, const char **argv)\n+{\n+\ttest_res = TEST(check_res = check_int(1, ==, 1), \"passing test\");\n+\tTEST(t_res(0), \"passing test and assertion return 0\");\n+\ttest_res = TEST(check_res = check_int(1, ==, 2), \"failing test\");\n+\tTEST(t_res(-1), \"failing test and assertion return -1\");\n+\ttest_res = TEST(t_todo(0), \"passing TEST_TODO()\");\n+\tTEST(t_res(0), \"passing TEST_TODO() returns 0\");\n+\ttest_res = TEST(t_todo(1), \"failing TEST_TODO()\");\n+\tTEST(t_res(-1), \"failing TEST_TODO() returns -1\");\n+\ttest_res = TEST(t_skip(), \"test_skip()\");\n+\tTEST(check_int(test_res, ==, 0), \"skipped test returns 0\");\n+\ttest_res = TEST(t_skip_todo(), \"test_skip() inside TEST_TODO()\");\n+\tTEST(t_res(0), \"test_skip() inside TEST_TODO() returns 0\");\n+\ttest_res = TEST(t_todo_after_fail(), \"TEST_TODO() after failing check\");\n+\tTEST(check_int(test_res, ==, -1), \"TEST_TODO() after failing check returns -1\");\n+\ttest_res = TEST(t_fail_after_todo(), \"failing check after TEST_TODO()\");\n+\tTEST(check_int(test_res, ==, -1), \"failing check after TEST_TODO() returns -1\");\n+\tTEST(t_messages(), \"messages from failing string and char comparison\");\n+\ttest_res = TEST(t_empty(), \"test with no checks\");\n+\tTEST(check_int(test_res, ==, -1), \"test with no checks returns -1\");\n+\n+\treturn test_done();\n+}\ndiff --git a/t/unit-tests/t-strbuf.c b/t/unit-tests/t-strbuf.c\nnew file mode 100644\nindex 0000000000..561611e242\n--- /dev/null\n+++ b/t/unit-tests/t-strbuf.c\n@@ -0,0 +1,75 @@\n+#include \"test-lib.h\"\n+#include \"strbuf.h\"\n+\n+/* wrapper that supplies tests with an initialized strbuf */\n+static void setup(void (*f)(struct strbuf*, void*), void *data)\n+{\n+\tstruct strbuf buf = STRBUF_INIT;\n+\n+\tf(&buf, data);\n+\tstrbuf_release(&buf);\n+\tcheck_uint(buf.len, ==, 0);\n+\tcheck_uint(buf.alloc, ==, 0);\n+\tcheck(buf.buf == strbuf_slopbuf);\n+\tcheck_char(buf.buf[0], ==, '\\0');\n+}\n+\n+static void t_static_init(void)\n+{\n+\tstruct strbuf buf = STRBUF_INIT;\n+\n+\tcheck_uint(buf.len, ==, 0);\n+\tcheck_uint(buf.alloc, ==, 0);\n+\tif (check(buf.buf == strbuf_slopbuf))\n+\t\treturn; /* avoid de-referencing buf.buf */\n+\tcheck_char(buf.buf[0], ==, '\\0');\n+}\n+\n+static void t_dynamic_init(void)\n+{\n+\tstruct strbuf buf;\n+\n+\tstrbuf_init(&buf, 1024);\n+\tcheck_uint(buf.len, ==, 0);\n+\tcheck_uint(buf.alloc, >=, 1024);\n+\tcheck_char(buf.buf[0], ==, '\\0');\n+\tstrbuf_release(&buf);\n+}\n+\n+static void t_addch(struct strbuf *buf, void *data)\n+{\n+\tconst char *p_ch = data;\n+\tconst char ch = *p_ch;\n+\n+\tstrbuf_addch(buf, ch);\n+\tif (check_uint(buf->len, ==, 1) ||\n+\t check_uint(buf->alloc, >, 1))\n+\t\treturn; /* avoid de-referencing buf->buf */\n+\tcheck_char(buf->buf[0], ==, ch);\n+\tcheck_char(buf->buf[1], ==, '\\0');\n+}\n+\n+static void t_addstr(struct strbuf *buf, void *data)\n+{\n+\tconst char *text = data;\n+\tsize_t len = strlen(text);\n+\n+\tstrbuf_addstr(buf, text);\n+\tif (check_uint(buf->len, ==, len) ||\n+\t check_uint(buf->alloc, >, len) ||\n+\t check_char(buf->buf[len], ==, '\\0'))\n+\t return;\n+\tcheck_str(buf->buf, text);\n+}\n+\n+int cmd_main(int argc, const char **argv)\n+{\n+\tif (TEST(t_static_init(), \"static initialization works\"))\n+\t\ttest_skip_all(\"STRBUF_INIT is broken\");\n+\tTEST(t_dynamic_init(), \"dynamic initialization works\");\n+\tTEST(setup(t_addch, \"a\"), \"strbuf_addch adds char\");\n+\tTEST(setup(t_addch, \"\"), \"strbuf_addch adds NUL char\");\n+\tTEST(setup(t_addstr, \"hello there\"), \"strbuf_addstr adds string\");\n+\n+\treturn test_done();\n+}\ndiff --git a/t/unit-tests/test-lib.c b/t/unit-tests/test-lib.c\nnew file mode 100644\nindex 0000000000..70030d587f\n--- /dev/null\n+++ b/t/unit-tests/test-lib.c\n@@ -0,0 +1,329 @@\n+#include \"test-lib.h\"\n+\n+enum result {\n+\tRESULT_NONE,\n+\tRESULT_FAILURE,\n+\tRESULT_SKIP,\n+\tRESULT_SUCCESS,\n+\tRESULT_TODO\n+};\n+\n+static struct {\n+\tenum result result;\n+\tint count;\n+\tunsigned failed :1;\n+\tunsigned lazy_plan :1;\n+\tunsigned running :1;\n+\tunsigned skip_all :1;\n+\tunsigned todo :1;\n+} ctx = {\n+\t.lazy_plan = 1,\n+\t.result = RESULT_NONE,\n+};\n+\n+static void msg_with_prefix(const char *prefix, const char *format, va_list ap)\n+{\n+\tfflush(stderr);\n+\tif (prefix)\n+\t\tfprintf(stdout, \"%s\", prefix);\n+\tvprintf(format, ap); /* TODO: handle newlines */\n+\tputc('\\n', stdout);\n+\tfflush(stdout);\n+}\n+\n+void test_msg(const char *format, ...)\n+{\n+\tva_list ap;\n+\n+\tva_start(ap, format);\n+\tmsg_with_prefix(\"# \", format, ap);\n+\tva_end(ap);\n+}\n+\n+void test_plan(int count)\n+{\n+\tassert(!ctx.running);\n+\n+\tfflush(stderr);\n+\tprintf(\"1..%d\\n\", count);\n+\tfflush(stdout);\n+\tctx.lazy_plan = 0;\n+}\n+\n+int test_done(void)\n+{\n+\tassert(!ctx.running);\n+\n+\tif (ctx.lazy_plan)\n+\t\ttest_plan(ctx.count);\n+\n+\treturn ctx.failed;\n+}\n+\n+void test_skip(const char *format, ...)\n+{\n+\tva_list ap;\n+\n+\tassert(ctx.running);\n+\n+\tctx.result = RESULT_SKIP;\n+\tva_start(ap, format);\n+\tif (format)\n+\t\tmsg_with_prefix(\"# skipping test - \", format, ap);\n+\tva_end(ap);\n+}\n+\n+void test_skip_all(const char *format, ...)\n+{\n+\tva_list ap;\n+\tconst char *prefix;\n+\n+\tif (!ctx.count && ctx.lazy_plan) {\n+\t\t/* We have not printed a test plan yet */\n+\t\tprefix = \"1..0 # SKIP \";\n+\t\tctx.lazy_plan = 0;\n+\t} else {\n+\t\t/* We have already printed a test plan */\n+\t\tprefix = \"Bail out! # \";\n+\t\tctx.failed = 1;\n+\t}\n+\tctx.skip_all = 1;\n+\tctx.result = RESULT_SKIP;\n+\tva_start(ap, format);\n+\tmsg_with_prefix(prefix, format, ap);\n+\tva_end(ap);\n+}\n+\n+int test__run_begin(void)\n+{\n+\tassert(!ctx.running);\n+\n+\tctx.count++;\n+\tctx.result = RESULT_NONE;\n+\tctx.running = 1;\n+\n+\treturn ctx.skip_all;\n+}\n+\n+static void print_description(const char *format, va_list ap)\n+{\n+\tif (format) {\n+\t\tfputs(\" - \", stdout);\n+\t\tvprintf(format, ap);\n+\t}\n+}\n+\n+int test__run_end(int was_run UNUSED, const char *location, const char *format, ...)\n+{\n+\tva_list ap;\n+\n+\tassert(ctx.running);\n+\tassert(!ctx.todo);\n+\n+\tfflush(stderr);\n+\tva_start(ap, format);\n+\tif (!ctx.skip_all) {\n+\t\tswitch (ctx.result) {\n+\t\tcase RESULT_SUCCESS:\n+\t\t\tprintf(\"ok %d\", ctx.count);\n+\t\t\tprint_description(format, ap);\n+\t\t\tbreak;\n+\n+\t\tcase RESULT_FAILURE:\n+\t\t\tprintf(\"not ok %d\", ctx.count);\n+\t\t\tprint_description(format, ap);\n+\t\t\tbreak;\n+\n+\t\tcase RESULT_TODO:\n+\t\t\tprintf(\"not ok %d\", ctx.count);\n+\t\t\tprint_description(format, ap);\n+\t\t\tprintf(\" # TODO\");\n+\t\t\tbreak;\n+\n+\t\tcase RESULT_SKIP:\n+\t\t\tprintf(\"ok %d\", ctx.count);\n+\t\t\tprint_description(format, ap);\n+\t\t\tprintf(\" # SKIP\");\n+\t\t\tbreak;\n+\n+\t\tcase RESULT_NONE:\n+\t\t\ttest_msg(\"BUG: test has no checks at %s\", location);\n+\t\t\tprintf(\"not ok %d\", ctx.count);\n+\t\t\tprint_description(format, ap);\n+\t\t\tctx.result = RESULT_FAILURE;\n+\t\t\tbreak;\n+\t\t}\n+\t}\n+\tva_end(ap);\n+\tctx.running = 0;\n+\tif (ctx.skip_all)\n+\t\treturn 0;\n+\tputc('\\n', stdout);\n+\tfflush(stdout);\n+\tctx.failed |= ctx.result == RESULT_FAILURE;\n+\n+\treturn -(ctx.result == RESULT_FAILURE);\n+}\n+\n+static void test_fail(void)\n+{\n+\tassert(ctx.result != RESULT_SKIP);\n+\n+\tctx.result = RESULT_FAILURE;\n+}\n+\n+static void test_pass(void)\n+{\n+\tassert(ctx.result != RESULT_SKIP);\n+\n+\tif (ctx.result == RESULT_NONE)\n+\t\tctx.result = RESULT_SUCCESS;\n+}\n+\n+static void test_todo(void)\n+{\n+\tassert(ctx.result != RESULT_SKIP);\n+\n+\tif (ctx.result != RESULT_FAILURE)\n+\t\tctx.result = RESULT_TODO;\n+}\n+\n+int test_assert(const char *location, const char *check, int ok)\n+{\n+\tassert(ctx.running);\n+\n+\tif (ctx.result == RESULT_SKIP) {\n+\t\ttest_msg(\"skipping check '%s' at %s\", check, location);\n+\t\treturn 0;\n+\t} else if (!ctx.todo) {\n+\t\tif (ok) {\n+\t\t\ttest_pass();\n+\t\t} else {\n+\t\t\ttest_msg(\"check \\\"%s\\\" failed at %s\", check, location);\n+\t\t\ttest_fail();\n+\t\t}\n+\t}\n+\n+\treturn -!ok;\n+}\n+\n+void test__todo_begin(void)\n+{\n+\tassert(ctx.running);\n+\tassert(!ctx.todo);\n+\n+\tctx.todo = 1;\n+}\n+\n+int test__todo_end(const char *location, const char *check, int res)\n+{\n+\tassert(ctx.running);\n+\tassert(ctx.todo);\n+\n+\tctx.todo = 0;\n+\tif (ctx.result == RESULT_SKIP)\n+\t\treturn 0;\n+\tif (!res) {\n+\t\ttest_msg(\"todo check '%s' succeeded at %s\", check, location);\n+\t\ttest_fail();\n+\t} else {\n+\t\ttest_todo();\n+\t}\n+\n+\treturn -!res;\n+}\n+\n+int check_bool_loc(const char *loc, const char *check, int ok)\n+{\n+\treturn test_assert(loc, check, ok);\n+}\n+\n+union test__tmp test__tmp[2];\n+\n+int check_int_loc(const char *loc, const char *check, int ok,\n+\t\t intmax_t a, intmax_t b)\n+{\n+\tint ret = test_assert(loc, check, ok);\n+\n+\tif (ret) {\n+\t\ttest_msg(\" left: %\"PRIdMAX, a);\n+\t\ttest_msg(\" right: %\"PRIdMAX, b);\n+\t}\n+\n+\treturn ret;\n+}\n+\n+int check_uint_loc(const char *loc, const char *check, int ok,\n+\t\t uintmax_t a, uintmax_t b)\n+{\n+\tint ret = test_assert(loc, check, ok);\n+\n+\tif (ret) {\n+\t\ttest_msg(\" left: %\"PRIuMAX, a);\n+\t\ttest_msg(\" right: %\"PRIuMAX, b);\n+\t}\n+\n+\treturn ret;\n+}\n+\n+static void print_one_char(char ch, char quote)\n+{\n+\tif ((unsigned char)ch < 0x20u || ch == 0x7f) {\n+\t\t/* TODO: improve handling of \\a, \\b, \\f ... */\n+\t\tprintf(\"\\\\%03o\", (unsigned char)ch);\n+\t} else {\n+\t\tif (ch == '\\\\' || ch == quote)\n+\t\t\tputc('\\\\', stdout);\n+\t\tputc(ch, stdout);\n+\t}\n+}\n+\n+static void print_char(const char *prefix, char ch)\n+{\n+\tprintf(\"# %s: '\", prefix);\n+\tprint_one_char(ch, '\\'');\n+\tfputs(\"'\\n\", stdout);\n+}\n+\n+int check_char_loc(const char *loc, const char *check, int ok, char a, char b)\n+{\n+\tint ret = test_assert(loc, check, ok);\n+\n+\tif (ret) {\n+\t\tfflush(stderr);\n+\t\tprint_char(\" left\", a);\n+\t\tprint_char(\" right\", b);\n+\t\tfflush(stdout);\n+\t}\n+\n+\treturn ret;\n+}\n+\n+static void print_str(const char *prefix, const char *str)\n+{\n+\tprintf(\"# %s: \", prefix);\n+\tif (!str) {\n+\t\tfputs(\"NULL\\n\", stdout);\n+\t} else {\n+\t\tputc('\"', stdout);\n+\t\twhile (*str)\n+\t\t\tprint_one_char(*str++, '\"');\n+\t\tfputs(\"\\\"\\n\", stdout);\n+\t}\n+}\n+\n+int check_str_loc(const char *loc, const char *check,\n+\t\t const char *a, const char *b)\n+{\n+\tint ok = (!a && !b) || (a && b && !strcmp(a, b));\n+\tint ret = test_assert(loc, check, ok);\n+\n+\tif (ret) {\n+\t\tfflush(stderr);\n+\t\tprint_str(\" left\", a);\n+\t\tprint_str(\" right\", b);\n+\t\tfflush(stdout);\n+\t}\n+\n+\treturn ret;\n+}\ndiff --git a/t/unit-tests/test-lib.h b/t/unit-tests/test-lib.h\nnew file mode 100644\nindex 0000000000..720c97c6f8\n--- /dev/null\n+++ b/t/unit-tests/test-lib.h\n@@ -0,0 +1,143 @@\n+#ifndef TEST_LIB_H\n+#define TEST_LIB_H\n+\n+#include \"git-compat-util.h\"\n+\n+/*\n+ * Run a test function, returns 0 if the test succeeds, -1 if it\n+ * fails. If test_skip_all() has been called then the test will not be\n+ * run. The description for each test should be unique. For example:\n+ *\n+ * TEST(test_something(arg1, arg2), \"something %d %d\", arg1, arg2)\n+ */\n+#define TEST(t, ...)\t\t\t\t\t\\\n+\ttest__run_end(test__run_begin() ? 0 : (t, 1),\t\\\n+\t\t TEST_LOCATION(), __VA_ARGS__)\n+\n+/*\n+ * Print a test plan, should be called before any tests. If the number\n+ * of tests is not known in advance test_done() will automatically\n+ * print a plan at the end of the test program.\n+ */\n+void test_plan(int count);\n+\n+/*\n+ * test_done() must be called at the end of main(). It will print the\n+ * plan if plan() was not called at the beginning of the test program\n+ * and returns the exit code for the test program.\n+ */\n+int test_done(void);\n+\n+/* Skip the current test. */\n+__attribute__((format (printf, 1, 2)))\n+void test_skip(const char *format, ...);\n+\n+/* Skip all remaining tests. */\n+__attribute__((format (printf, 1, 2)))\n+void test_skip_all(const char *format, ...);\n+\n+/* Print a diagnostic message to stdout. */\n+__attribute__((format (printf, 1, 2)))\n+void test_msg(const char *format, ...);\n+\n+/*\n+ * Test checks are built around test_assert(). checks return 0 on\n+ * success, -1 on failure. If any check fails then the test will\n+ * fail. To create a custom check define a function that wraps\n+ * test_assert() and a macro to wrap that function. For example:\n+ *\n+ * static int check_oid_loc(const char *loc, const char *check,\n+ *\t\t\t struct object_id *a, struct object_id *b)\n+ * {\n+ *\t int res = test_assert(loc, check, oideq(a, b));\n+ *\n+ *\t if (res) {\n+ *\t\t test_msg(\" left: %s\", oid_to_hex(a);\n+ *\t\t test_msg(\" right: %s\", oid_to_hex(a);\n+ *\n+ *\t }\n+ *\t return res;\n+ * }\n+ *\n+ * #define check_oid(a, b) \\\n+ *\t check_oid_loc(TEST_LOCATION(), \"oideq(\"#a\", \"#b\")\", a, b)\n+ */\n+int test_assert(const char *location, const char *check, int ok);\n+\n+/* Helper macro to pass the location to checks */\n+#define TEST_LOCATION() TEST__MAKE_LOCATION(__LINE__)\n+\n+/* Check a boolean condition. */\n+#define check(x)\t\t\t\t\\\n+\tcheck_bool_loc(TEST_LOCATION(), #x, x)\n+int check_bool_loc(const char *loc, const char *check, int ok);\n+\n+/*\n+ * Compare two integers. Prints a message with the two values if the\n+ * comparison fails. NB this is not thread safe.\n+ */\n+#define check_int(a, op, b)\t\t\t\t\t\t\\\n+\t(test__tmp[0].i = (a), test__tmp[1].i = (b),\t\t\t\\\n+\t check_int_loc(TEST_LOCATION(), #a\" \"#op\" \"#b,\t\t\t\\\n+\t\t test__tmp[0].i op test__tmp[1].i, a, b))\n+int check_int_loc(const char *loc, const char *check, int ok,\n+\t\t intmax_t a, intmax_t b);\n+\n+/*\n+ * Compare two unsigned integers. Prints a message with the two values\n+ * if the comparison fails. NB this is not thread safe.\n+ */\n+#define check_uint(a, op, b)\t\t\t\t\t\t\\\n+\t(test__tmp[0].u = (a), test__tmp[1].u = (b),\t\t\t\\\n+\t check_uint_loc(TEST_LOCATION(), #a\" \"#op\" \"#b,\t\t\t\\\n+\t\t\ttest__tmp[0].u op test__tmp[1].u, a, b))\n+int check_uint_loc(const char *loc, const char *check, int ok,\n+\t\t uintmax_t a, uintmax_t b);\n+\n+/*\n+ * Compare two chars. Prints a message with the two values if the\n+ * comparison fails. NB this is not thread safe.\n+ */\n+#define check_char(a, op, b)\t\t\t\t\t\t\\\n+\t(test__tmp[0].c = (a), test__tmp[1].c = (b),\t\t\t\\\n+\t check_char_loc(TEST_LOCATION(), #a\" \"#op\" \"#b,\t\t\t\\\n+\t\t\ttest__tmp[0].c op test__tmp[1].c, a, b))\n+int check_char_loc(const char *loc, const char *check, int ok,\n+\t\t char a, char b);\n+\n+/* Check whether two strings are equal. */\n+#define check_str(a, b)\t\t\t\t\t\t\t\\\n+\tcheck_str_loc(TEST_LOCATION(), \"!strcmp(\"#a\", \"#b\")\", a, b)\n+int check_str_loc(const char *loc, const char *check,\n+\t\t const char *a, const char *b);\n+\n+/*\n+ * Wrap a check that is known to fail. If the check succeeds then the\n+ * test will fail. Returns 0 if the check fails, -1 if it\n+ * succeeds. For example:\n+ *\n+ * TEST_TODO(check(0));\n+ */\n+#define TEST_TODO(check) \\\n+\t(test__todo_begin(), test__todo_end(TEST_LOCATION(), #check, check))\n+\n+/* Private helpers */\n+\n+#define TEST__STR(x) #x\n+#define TEST__MAKE_LOCATION(line) __FILE__ \":\" TEST__STR(line)\n+\n+union test__tmp {\n+\tintmax_t i;\n+\tuintmax_t u;\n+\tchar c;\n+};\n+\n+extern union test__tmp test__tmp[2];\n+\n+int test__run_begin(void);\n+__attribute__((format (printf, 3, 4)))\n+int test__run_end(int, const char *, const char *, ...);\n+void test__todo_begin(void);\n+int test__todo_end(const char *, const char *, int);\n+\n+#endif /* TEST_LIB_H */\n-- \n2.42.0.rc1.204.g551eb34607-goog\n\n"},{"id":"480766","messageId":"xmqqo7j5be1b.fsf@gitster.g","threadId":"59937","inReplyTo":"cover.1692297001.git.steadmon@google.com","subject":"Re: [PATCH v7 0/3] Add unit test framework and project plan","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2023-08-17T20:38:08Z","receivedAt":"2023-08-17T20:39:06Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Josh Steadmon writes:\n\n> Changes in v7:\n> - Fix corrupt diff in patch #2, sorry for the noise.\n\nThanks for resending. Will queue.\n"},{"id":"480778","messageId":"xmqq350hw6n7.fsf@gitster.g","threadId":"59937","inReplyTo":"3cc98d4045eeda6e8cc24914802edc16d367fba0.1692297001.git.steadmon@google.com","subject":"Re: [PATCH v7 2/3] unit tests: add TAP unit test framework","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2023-08-18T00:12:12Z","receivedAt":"2023-08-18T00:14:00Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Josh Steadmon writes:\n\n> +test_expect_success 'TAP output from unit tests' '\n> +\tcat >expect <<-EOF &&\n> +\tok 1 - passing test\n> +\tok 2 - passing test and assertion return 0\n> +\t# check \"1 == 2\" failed at t/unit-tests/t-basic.c:76\n> +\t# left: 1\n> +\t# right: 2\n> +\tnot ok 3 - failing test\n> +\tok 4 - failing test and assertion return -1\n> +\tnot ok 5 - passing TEST_TODO() # TODO\n> +\tok 6 - passing TEST_TODO() returns 0\n> +\t# todo check ${SQ}check(x)${SQ} succeeded at t/unit-tests/t-basic.c:25\n> +\tnot ok 7 - failing TEST_TODO()\n> +\tok 8 - failing TEST_TODO() returns -1\n> +\t# check \"0\" failed at t/unit-tests/t-basic.c:30\n> +\t# skipping test - missing prerequisite\n> +\t# skipping check ${SQ}1${SQ} at t/unit-tests/t-basic.c:32\n> +\tok 9 - test_skip() # SKIP\n> +\tok 10 - skipped test returns 0\n> +\t# skipping test - missing prerequisite\n> +\tok 11 - test_skip() inside TEST_TODO() # SKIP\n> +\tok 12 - test_skip() inside TEST_TODO() returns 0\n> +\t# check \"0\" failed at t/unit-tests/t-basic.c:48\n> +\tnot ok 13 - TEST_TODO() after failing check\n> +\tok 14 - TEST_TODO() after failing check returns -1\n> +\t# check \"0\" failed at t/unit-tests/t-basic.c:56\n> +\tnot ok 15 - failing check after TEST_TODO()\n> +\tok 16 - failing check after TEST_TODO() returns -1\n> +\t# check \"!strcmp(\"\\thello\\\\\\\\\", \"there\\\"\\n\")\" failed at t/unit-tests/t-basic.c:61\n> +\t# left: \"\\011hello\\\\\\\\\"\n> +\t# right: \"there\\\"\\012\"\n> +\t# check \"!strcmp(\"NULL\", NULL)\" failed at t/unit-tests/t-basic.c:62\n> +\t# left: \"NULL\"\n> +\t# right: NULL\n> +\t# check \"${SQ}a${SQ} == ${SQ}\\n${SQ}\" failed at t/unit-tests/t-basic.c:63\n> +\t# left: ${SQ}a${SQ}\n> +\t# right: ${SQ}\\012${SQ}\n> +\t# check \"${SQ}\\\\\\\\${SQ} == ${SQ}\\\\${SQ}${SQ}\" failed at t/unit-tests/t-basic.c:64\n> +\t# left: ${SQ}\\\\\\\\${SQ}\n> +\t# right: ${SQ}\\\\${SQ}${SQ}\n> +\tnot ok 17 - messages from failing string and char comparison\n> +\t# BUG: test has no checks at t/unit-tests/t-basic.c:91\n> +\tnot ok 18 - test with no checks\n> +\tok 19 - test with no checks returns -1\n> +\t1..19\n> +\tEOF\n\nPresumably t-basic will serve as a catalog of check_* functions and\nthe test binary, together with this test piece, will keep growing as\nwe gain features in the unit tests infrastructure. I wonder how\nmaintainable the above is, though. When we acquire new test, we\nwould need to renumber. What if multiple developers add new\nfeatures to the catalog at the same time?\n\n> diff --git a/t/unit-tests/.gitignore b/t/unit-tests/.gitignore\n> new file mode 100644\n> index 0000000000..e292d58348\n> --- /dev/null\n> +++ b/t/unit-tests/.gitignore\n> @@ -0,0 +1,2 @@\n> +/t-basic\n> +/t-strbuf\n\nAlso, can we come up with some naming convention so that we do not\nhave to keep adding to this file every time we add a new test\nscript?\n\n> diff --git a/t/unit-tests/t-strbuf.c b/t/unit-tests/t-strbuf.c\n> new file mode 100644\n> index 0000000000..561611e242\n> --- /dev/null\n> +++ b/t/unit-tests/t-strbuf.c\n> @@ -0,0 +1,75 @@\n> +#include \"test-lib.h\"\n> +#include \"strbuf.h\"\n> +\n> +/* wrapper that supplies tests with an initialized strbuf */\n> +static void setup(void (*f)(struct strbuf*, void*), void *data)\n> +{\n> +\tstruct strbuf buf = STRBUF_INIT;\n> +\n> +\tf(&buf, data);\n> +\tstrbuf_release(&buf);\n> +\tcheck_uint(buf.len, ==, 0);\n> +\tcheck_uint(buf.alloc, ==, 0);\n> +\tcheck(buf.buf == strbuf_slopbuf);\n> +\tcheck_char(buf.buf[0], ==, '\\0');\n> +}\n\nWhat I am going to utter from here on are not complaints but purely\nmeant as questions. \n\nWould the resulting output and maintainability of the tests change\n(improve, or worsen) if we introduce\n\n\tstatic void assert_empty_strbuf(struct strbuf *buf)\n\t{\n\t\tcheck_uint(buf->len, ==, 0);\n check_uint(buf->alloc, ==, 0);\n\t\tcheck(buf.buf == strbuf_slopbuf);\n\t\tcheck_char(buf.buf[0], ==, '\\0');\n\t}\n\nand call it from the setup() function to ensure that\nstrbuf_release(&buf) it calls after running customer test f() brings\nthe buffer in a reasonably initialized state? The t_static_init()\ntest should be able to say\n\n\tstatic void t_static_init(void)\n\t{\n\t\tstruct strbuf buf = STRBUF_INIT;\n\t\tassert_empty_strbuf(&buf);\n\t}\n\nif we did so, but is that a good thing or a bad thing (e.g. it may\nmake it harder to figure out where the real error came from, because\nof the \"line number\" thing will not easily capture the caller of the\ncaller, perhaps)? \n\n> +static void t_static_init(void)\n> +{\n> +\tstruct strbuf buf = STRBUF_INIT;\n> +\n> +\tcheck_uint(buf.len, ==, 0);\n> +\tcheck_uint(buf.alloc, ==, 0);\n> +\tif (check(buf.buf == strbuf_slopbuf))\n> +\t\treturn; /* avoid de-referencing buf.buf */\n\nstrbuf_slopbuf[0] is designed to be readable. Do check() assertions\nreturn their parameter negated?\n\nIn other words, if \"we expect buf.buf to point at the slopbuf, but\nif that expectation does not hold, check() returns true and we\nrefrain from doing check_char() on the next line because we cannot\ntrust what buf.buf points at\" is what is going on here, I find it\nvery confusing. Perhaps my intuition is failing me, but somehow I\nwould have expected that passing check_foo() would return true while\nfailing ones would return false.\n\nIOW I would expect\n\n\tif (check(buf.buf == strbuf_slopbuf))\n\t\treturn;\n\nto work very similarly to\n\n\tif (buf.buf == strbuf_slopbuf)\n\t\treturn;\n\nin expressing the control flow, simply because they are visually\nsimilar. But of course, if we early-return because buf.buf that\ndoes not point at strbuf_slopbuf is a sign of trouble, then the\ncontrol flow we want is\n\n\tif (buf.buf != strbuf_slopbuf)\n\t\treturn;\n\nor\n\n\tif (!(buf.buf == strbuf_slopbuf))\n\t\treturn;\n\nThe latter is easier to translate to check_foo(), because what is\ninside the inner parentheses is the condition we expect, and we\nwould like check_foo() to complain when the condition does not hold.\n\nFor the \"check_foo()\" thing to work in a similar way, while having\nthe side effect of reporting any failed expectations, we would want\nto write\n\n\tif (!check(buf.buf == strbuf_slopbuf))\n\t\treturn;\n\nAnd for that similarity to work, check_foo() must return false when\nits expectation fails, and return true when its expectation holds.\n\nI think that is where my \"I find it very confusing\" comes from.\n\n> +\tcheck_char(buf.buf[0], ==, '\\0');\n> +}\n\n> +static void t_dynamic_init(void)\n> +{\n> +\tstruct strbuf buf;\n> +\n> +\tstrbuf_init(&buf, 1024);\n> +\tcheck_uint(buf.len, ==, 0);\n> +\tcheck_uint(buf.alloc, >=, 1024);\n> +\tcheck_char(buf.buf[0], ==, '\\0');\n\nIs it sensible to check buf.buf is not slopbuf at this point, or\ndoes it make the test TOO intimate with the current implementation\ndetail?\n\n> +\tstrbuf_release(&buf);\n> +}\n> +\n> +static void t_addch(struct strbuf *buf, void *data)\n> +{\n> +\tconst char *p_ch = data;\n> +\tconst char ch = *p_ch;\n> +\n> +\tstrbuf_addch(buf, ch);\n> +\tif (check_uint(buf->len, ==, 1) ||\n> +\t check_uint(buf->alloc, >, 1))\n> +\t\treturn; /* avoid de-referencing buf->buf */\n\nAgain, I find the return values from these check_uint() calls highly\nconfusing, if this is saying \"if len is 1 and alloc is more than 1,\nthen we are in an expected state and can further validate that buf[0]\nis ch and buf[1] is NULL, but otherwise we should punt\". The polarity\nlooks screwy. Perhaps it is just me?\n\n> +\tcheck_char(buf->buf[0], ==, ch);\n> +\tcheck_char(buf->buf[1], ==, '\\0');\n> +}\n\nIn any case, this t_addch() REQUIRES that incoming buf is empty,\ndoesn't it? I do not think it is sensible. I would have expected\nthat it would be more like\n\n\tt_addch(struct strbuf *buf, void *data)\n\t{\n\t\tchar ch = *(char *)data;\n\t\tsize_t orig_alloc = buf->alloc;\n\t\tsize_t orig_len = buf->len;\n\n\t\tif (!assert_sane_strbuf(buf))\n\t\t\treturn;\n strbuf_addch(buf, ch);\n\t\tif (!assert_sane_strbuf(buf))\n\t\t\treturn;\n\t\tcheck_uint(buf->len, ==, orig_len + 1);\n\t\tcheck_uint(buf->alloc, >=, orig_alloc);\n check_char(buf->buf[buf->len - 1], ==, ch);\n check_char(buf->buf[buf->len], ==, '\\0');\n\t}\n\nto ensure that we can add a ch to a strbuf with any existing\ncontents and get a one-byte longer contents than before, with the\nlast byte of the buffer becoming 'ch' and still NUL terminated.\n\nAnd we protect ourselves with a helper that checks if the given\nstrbuf looks *sane*.\n\n\tstatic int assert_sane_strbuf(struct strbuf *buf)\n\t{\n \t/* can use slopbuf only when the length is 0 */\n\t\tif (buf->buf == strbuf_slopbuf)\n \treturn (buf->len == 0);\n\t\t/* everybody else must have non-NULL buffer */\n\t\tif (buf->buf == NULL)\n\t\t\treturn 0;\n /* \n\t\t * alloc must be at least 1 byte larger than len\n\t\t * for the terminating NUL at the end.\n\t\t */\n\t\treturn ((buf->len + 1 <= buf->alloc) &&\n\t\t \t(buf->buf[buf->len] == '\\0'));\n\t}\n\nYou can obviously use your check_foo() for the individual checks\ndone in this function to get a more detailed diagnosis, but because\nI have confused myself enough by thinking about their polarity, I\nwrote this in barebones comparison instead.\n\n"},{"id":"481011","messageId":"ZOe5gtYeQaKqOJVm@google.com","threadId":"59937","inReplyTo":"cover.1692297001.git.steadmon@google.com","subject":"Re: [PATCH v7 0/3] Add unit test framework and project plan","fromName":"Josh Steadmon","fromEmail":"steadmon@google.com","sentAt":"2023-08-24T20:11:46Z","receivedAt":"2023-08-24T20:12:38Z","isPatch":true,"sender":{"key":"steadmon@google.com","avatar":"https://avatars.githubusercontent.com/u/2654920?v=4"},"body":"BTW, I'm going to be AFK for a couple weeks, so it will be a while\nbefore I'm able to address feedback on this series. Thanks in advance.\n"},{"id":"481808","messageId":"xmqqttrykkji.fsf@gitster.g","threadId":"59937","inReplyTo":"ZOe5gtYeQaKqOJVm@google.com","subject":"Re: [PATCH v7 0/3] Add unit test framework and project plan","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2023-09-13T18:14:25Z","receivedAt":"2023-09-13T18:14:32Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Josh Steadmon writes:\n\n> BTW, I'm going to be AFK for a couple weeks, so it will be a while\n> before I'm able to address feedback on this series. Thanks in advance.\n\nOK. Enjoy your time off.\n"},{"id":"482181","messageId":"xmqqa5te0y9r.fsf@gitster.g","threadId":"59937","inReplyTo":"xmqq350hw6n7.fsf@gitster.g","subject":"Re: [PATCH v7 2/3] unit tests: add TAP unit test framework","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2023-09-22T20:05:20Z","receivedAt":"2023-09-22T20:05:28Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"It seems this got stuck during Josh's absense and I didn't ping it\nfurther, but I should have noticed that you are the author of this\npatch, and pinged you in the meantime.\n\nAny thought on the \"polarity\" of the return values from the\nassertion? I still find it confusing and hard to follow.\n\nThanks.\n\n> Josh Steadmon writes:\n>\n>> +test_expect_success 'TAP output from unit tests' '\n>> +\tcat >expect <<-EOF &&\n>> +\tok 1 - passing test\n>> +\tok 2 - passing test and assertion return 0\n>> +\t# check \"1 == 2\" failed at t/unit-tests/t-basic.c:76\n>> +\t# left: 1\n>> +\t# right: 2\n>> +\tnot ok 3 - failing test\n>> +\tok 4 - failing test and assertion return -1\n>> +\tnot ok 5 - passing TEST_TODO() # TODO\n>> +\tok 6 - passing TEST_TODO() returns 0\n>> +\t# todo check ${SQ}check(x)${SQ} succeeded at t/unit-tests/t-basic.c:25\n>> +\tnot ok 7 - failing TEST_TODO()\n>> +\tok 8 - failing TEST_TODO() returns -1\n>> +\t# check \"0\" failed at t/unit-tests/t-basic.c:30\n>> +\t# skipping test - missing prerequisite\n>> +\t# skipping check ${SQ}1${SQ} at t/unit-tests/t-basic.c:32\n>> +\tok 9 - test_skip() # SKIP\n>> +\tok 10 - skipped test returns 0\n>> +\t# skipping test - missing prerequisite\n>> +\tok 11 - test_skip() inside TEST_TODO() # SKIP\n>> +\tok 12 - test_skip() inside TEST_TODO() returns 0\n>> +\t# check \"0\" failed at t/unit-tests/t-basic.c:48\n>> +\tnot ok 13 - TEST_TODO() after failing check\n>> +\tok 14 - TEST_TODO() after failing check returns -1\n>> +\t# check \"0\" failed at t/unit-tests/t-basic.c:56\n>> +\tnot ok 15 - failing check after TEST_TODO()\n>> +\tok 16 - failing check after TEST_TODO() returns -1\n>> +\t# check \"!strcmp(\"\\thello\\\\\\\\\", \"there\\\"\\n\")\" failed at t/unit-tests/t-basic.c:61\n>> +\t# left: \"\\011hello\\\\\\\\\"\n>> +\t# right: \"there\\\"\\012\"\n>> +\t# check \"!strcmp(\"NULL\", NULL)\" failed at t/unit-tests/t-basic.c:62\n>> +\t# left: \"NULL\"\n>> +\t# right: NULL\n>> +\t# check \"${SQ}a${SQ} == ${SQ}\\n${SQ}\" failed at t/unit-tests/t-basic.c:63\n>> +\t# left: ${SQ}a${SQ}\n>> +\t# right: ${SQ}\\012${SQ}\n>> +\t# check \"${SQ}\\\\\\\\${SQ} == ${SQ}\\\\${SQ}${SQ}\" failed at t/unit-tests/t-basic.c:64\n>> +\t# left: ${SQ}\\\\\\\\${SQ}\n>> +\t# right: ${SQ}\\\\${SQ}${SQ}\n>> +\tnot ok 17 - messages from failing string and char comparison\n>> +\t# BUG: test has no checks at t/unit-tests/t-basic.c:91\n>> +\tnot ok 18 - test with no checks\n>> +\tok 19 - test with no checks returns -1\n>> +\t1..19\n>> +\tEOF\n>\n> Presumably t-basic will serve as a catalog of check_* functions and\n> the test binary, together with this test piece, will keep growing as\n> we gain features in the unit tests infrastructure. I wonder how\n> maintainable the above is, though. When we acquire new test, we\n> would need to renumber. What if multiple developers add new\n> features to the catalog at the same time?\n>\n>> diff --git a/t/unit-tests/.gitignore b/t/unit-tests/.gitignore\n>> new file mode 100644\n>> index 0000000000..e292d58348\n>> --- /dev/null\n>> +++ b/t/unit-tests/.gitignore\n>> @@ -0,0 +1,2 @@\n>> +/t-basic\n>> +/t-strbuf\n>\n> Also, can we come up with some naming convention so that we do not\n> have to keep adding to this file every time we add a new test\n> script?\n>\n>> diff --git a/t/unit-tests/t-strbuf.c b/t/unit-tests/t-strbuf.c\n>> new file mode 100644\n>> index 0000000000..561611e242\n>> --- /dev/null\n>> +++ b/t/unit-tests/t-strbuf.c\n>> @@ -0,0 +1,75 @@\n>> +#include \"test-lib.h\"\n>> +#include \"strbuf.h\"\n>> +\n>> +/* wrapper that supplies tests with an initialized strbuf */\n>> +static void setup(void (*f)(struct strbuf*, void*), void *data)\n>> +{\n>> +\tstruct strbuf buf = STRBUF_INIT;\n>> +\n>> +\tf(&buf, data);\n>> +\tstrbuf_release(&buf);\n>> +\tcheck_uint(buf.len, ==, 0);\n>> +\tcheck_uint(buf.alloc, ==, 0);\n>> +\tcheck(buf.buf == strbuf_slopbuf);\n>> +\tcheck_char(buf.buf[0], ==, '\\0');\n>> +}\n>\n> What I am going to utter from here on are not complaints but purely\n> meant as questions. \n>\n> Would the resulting output and maintainability of the tests change\n> (improve, or worsen) if we introduce\n>\n> \tstatic void assert_empty_strbuf(struct strbuf *buf)\n> \t{\n> \t\tcheck_uint(buf->len, ==, 0);\n> check_uint(buf->alloc, ==, 0);\n> \t\tcheck(buf.buf == strbuf_slopbuf);\n> \t\tcheck_char(buf.buf[0], ==, '\\0');\n> \t}\n>\n> and call it from the setup() function to ensure that\n> strbuf_release(&buf) it calls after running customer test f() brings\n> the buffer in a reasonably initialized state? The t_static_init()\n> test should be able to say\n>\n> \tstatic void t_static_init(void)\n> \t{\n> \t\tstruct strbuf buf = STRBUF_INIT;\n> \t\tassert_empty_strbuf(&buf);\n> \t}\n>\n> if we did so, but is that a good thing or a bad thing (e.g. it may\n> make it harder to figure out where the real error came from, because\n> of the \"line number\" thing will not easily capture the caller of the\n> caller, perhaps)? \n>\n>> +static void t_static_init(void)\n>> +{\n>> +\tstruct strbuf buf = STRBUF_INIT;\n>> +\n>> +\tcheck_uint(buf.len, ==, 0);\n>> +\tcheck_uint(buf.alloc, ==, 0);\n>> +\tif (check(buf.buf == strbuf_slopbuf))\n>> +\t\treturn; /* avoid de-referencing buf.buf */\n>\n> strbuf_slopbuf[0] is designed to be readable. Do check() assertions\n> return their parameter negated?\n>\n> In other words, if \"we expect buf.buf to point at the slopbuf, but\n> if that expectation does not hold, check() returns true and we\n> refrain from doing check_char() on the next line because we cannot\n> trust what buf.buf points at\" is what is going on here, I find it\n> very confusing. Perhaps my intuition is failing me, but somehow I\n> would have expected that passing check_foo() would return true while\n> failing ones would return false.\n>\n> IOW I would expect\n>\n> \tif (check(buf.buf == strbuf_slopbuf))\n> \t\treturn;\n>\n> to work very similarly to\n>\n> \tif (buf.buf == strbuf_slopbuf)\n> \t\treturn;\n>\n> in expressing the control flow, simply because they are visually\n> similar. But of course, if we early-return because buf.buf that\n> does not point at strbuf_slopbuf is a sign of trouble, then the\n> control flow we want is\n>\n> \tif (buf.buf != strbuf_slopbuf)\n> \t\treturn;\n>\n> or\n>\n> \tif (!(buf.buf == strbuf_slopbuf))\n> \t\treturn;\n>\n> The latter is easier to translate to check_foo(), because what is\n> inside the inner parentheses is the condition we expect, and we\n> would like check_foo() to complain when the condition does not hold.\n>\n> For the \"check_foo()\" thing to work in a similar way, while having\n> the side effect of reporting any failed expectations, we would want\n> to write\n>\n> \tif (!check(buf.buf == strbuf_slopbuf))\n> \t\treturn;\n>\n> And for that similarity to work, check_foo() must return false when\n> its expectation fails, and return true when its expectation holds.\n>\n> I think that is where my \"I find it very confusing\" comes from.\n>\n>> +\tcheck_char(buf.buf[0], ==, '\\0');\n>> +}\n>\n>> +static void t_dynamic_init(void)\n>> +{\n>> +\tstruct strbuf buf;\n>> +\n>> +\tstrbuf_init(&buf, 1024);\n>> +\tcheck_uint(buf.len, ==, 0);\n>> +\tcheck_uint(buf.alloc, >=, 1024);\n>> +\tcheck_char(buf.buf[0], ==, '\\0');\n>\n> Is it sensible to check buf.buf is not slopbuf at this point, or\n> does it make the test TOO intimate with the current implementation\n> detail?\n>\n>> +\tstrbuf_release(&buf);\n>> +}\n>> +\n>> +static void t_addch(struct strbuf *buf, void *data)\n>> +{\n>> +\tconst char *p_ch = data;\n>> +\tconst char ch = *p_ch;\n>> +\n>> +\tstrbuf_addch(buf, ch);\n>> +\tif (check_uint(buf->len, ==, 1) ||\n>> +\t check_uint(buf->alloc, >, 1))\n>> +\t\treturn; /* avoid de-referencing buf->buf */\n>\n> Again, I find the return values from these check_uint() calls highly\n> confusing, if this is saying \"if len is 1 and alloc is more than 1,\n> then we are in an expected state and can further validate that buf[0]\n> is ch and buf[1] is NULL, but otherwise we should punt\". The polarity\n> looks screwy. Perhaps it is just me?\n>\n>> +\tcheck_char(buf->buf[0], ==, ch);\n>> +\tcheck_char(buf->buf[1], ==, '\\0');\n>> +}\n>\n> In any case, this t_addch() REQUIRES that incoming buf is empty,\n> doesn't it? I do not think it is sensible. I would have expected\n> that it would be more like\n>\n> \tt_addch(struct strbuf *buf, void *data)\n> \t{\n> \t\tchar ch = *(char *)data;\n> \t\tsize_t orig_alloc = buf->alloc;\n> \t\tsize_t orig_len = buf->len;\n>\n> \t\tif (!assert_sane_strbuf(buf))\n> \t\t\treturn;\n> strbuf_addch(buf, ch);\n> \t\tif (!assert_sane_strbuf(buf))\n> \t\t\treturn;\n> \t\tcheck_uint(buf->len, ==, orig_len + 1);\n> \t\tcheck_uint(buf->alloc, >=, orig_alloc);\n> check_char(buf->buf[buf->len - 1], ==, ch);\n> check_char(buf->buf[buf->len], ==, '\\0');\n> \t}\n>\n> to ensure that we can add a ch to a strbuf with any existing\n> contents and get a one-byte longer contents than before, with the\n> last byte of the buffer becoming 'ch' and still NUL terminated.\n>\n> And we protect ourselves with a helper that checks if the given\n> strbuf looks *sane*.\n>\n> \tstatic int assert_sane_strbuf(struct strbuf *buf)\n> \t{\n> \t/* can use slopbuf only when the length is 0 */\n> \t\tif (buf->buf == strbuf_slopbuf)\n> \treturn (buf->len == 0);\n> \t\t/* everybody else must have non-NULL buffer */\n> \t\tif (buf->buf == NULL)\n> \t\t\treturn 0;\n> /* \n> \t\t * alloc must be at least 1 byte larger than len\n> \t\t * for the terminating NUL at the end.\n> \t\t */\n> \t\treturn ((buf->len + 1 <= buf->alloc) &&\n> \t\t \t(buf->buf[buf->len] == '\\0'));\n> \t}\n>\n> You can obviously use your check_foo() for the individual checks\n> done in this function to get a more detailed diagnosis, but because\n> I have confused myself enough by thinking about their polarity, I\n> wrote this in barebones comparison instead.\n"},{"id":"482228","messageId":"0b6de919-8dbf-454f-807b-5abb64388cb7@gmail.com","threadId":"59937","inReplyTo":"xmqqa5te0y9r.fsf@gitster.g","subject":"Re: [PATCH v7 2/3] unit tests: add TAP unit test framework","fromName":"","fromEmail":"phillip.wood123@gmail.com","sentAt":"2023-09-24T13:57:53Z","receivedAt":"2023-09-24T13:58:03Z","isPatch":true,"sender":{"key":"phillip.wood@dunelm.org.uk","avatar":null},"body":"Hi Junio\n\nOn 22/09/2023 21:05, Junio C Hamano wrote:\n> It seems this got stuck during Josh's absense and I didn't ping it\n> further, but I should have noticed that you are the author of this\n> patch, and pinged you in the meantime.\n\nSorry I meant to reply when I saw your first message but then didn't get \nround to it.\n\n> Any thought on the \"polarity\" of the return values from the\n> assertion? I still find it confusing and hard to follow.\n\nWhen I was writing this I was torn between whether to follow our usual \nconvention of returning zero for success and minus one for failure or to \nreturn one for success and zero for failure. In the end I decided to go \nwith the former but I tend to agree with you that the latter would be \neasier to understand.\n\n>>> +test_expect_success 'TAP output from unit tests' '\n>>> [...]\n>>> +\tok 19 - test with no checks returns -1\n>>> +\t1..19\n>>> +\tEOF\n>>\n>> Presumably t-basic will serve as a catalog of check_* functions and\n>> the test binary, together with this test piece, will keep growing as\n>> we gain features in the unit tests infrastructure. I wonder how\n>> maintainable the above is, though. When we acquire new test, we\n>> would need to renumber. What if multiple developers add new\n>> features to the catalog at the same time?\n\nI think we could just add new tests to the end so we'd only need to \nchange the \"1..19\" line. That will become a source of merge conflicts if \nmultiple developers add new features at the same time though. Having \nseveral unit test programs called from separate tests in t0080 might \nhelp with that.\n\n>>> diff --git a/t/unit-tests/.gitignore b/t/unit-tests/.gitignore\n>>> new file mode 100644\n>>> index 0000000000..e292d58348\n>>> --- /dev/null\n>>> +++ b/t/unit-tests/.gitignore\n>>> @@ -0,0 +1,2 @@\n>>> +/t-basic\n>>> +/t-strbuf\n>>\n>> Also, can we come up with some naming convention so that we do not\n>> have to keep adding to this file every time we add a new test\n>> script?\n\nPerhaps we should put the unit test binaries in a separate directory so \nwe can just add that directory to .gitignore.\n\nBest Wishes\n\nPhillip\n"},{"id":"482298","messageId":"xmqqbkdqt70q.fsf@gitster.g","threadId":"59937","inReplyTo":"0b6de919-8dbf-454f-807b-5abb64388cb7@gmail.com","subject":"Re: [PATCH v7 2/3] unit tests: add TAP unit test framework","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2023-09-25T18:57:57Z","receivedAt":"2023-09-25T18:58:05Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"phillip.wood123@gmail.com writes:\n\n> When I was writing this I was torn between whether to follow our usual\n> convention of returning zero for success and minus one for failure or\n> to return one for success and zero for failure. In the end I decided\n> to go with the former but I tend to agree with you that the latter\n> would be easier to understand.\n\nAn understandable contention.\n\n>>>> @@ -0,0 +1,2 @@\n>>>> +/t-basic\n>>>> +/t-strbuf\n>>>\n>>> Also, can we come up with some naming convention so that we do not\n>>> have to keep adding to this file every time we add a new test\n>>> script?\n>\n> Perhaps we should put the unit test binaries in a separate directory\n> so we can just add that directory to .gitignore.\n\nYeah, if we can do that, that would help organizing these tests.\n\nThanks for working on this.\n\n"},{"id":"482775","messageId":"ZSCRFNkzXZb3fBaU@google.com","threadId":"59937","inReplyTo":"0b6de919-8dbf-454f-807b-5abb64388cb7@gmail.com","subject":"Re: [PATCH v7 2/3] unit tests: add TAP unit test framework","fromName":"Josh Steadmon","fromEmail":"steadmon@google.com","sentAt":"2023-10-06T22:58:28Z","receivedAt":"2023-10-06T22:58:37Z","isPatch":true,"sender":{"key":"steadmon@google.com","avatar":"https://avatars.githubusercontent.com/u/2654920?v=4"},"body":"On 2023.09.24 14:57, phillip.wood123@gmail.com wrote:\n> On 22/09/2023 21:05, Junio C Hamano wrote:\n> > Any thought on the \"polarity\" of the return values from the\n> > assertion? I still find it confusing and hard to follow.\n> \n> When I was writing this I was torn between whether to follow our usual\n> convention of returning zero for success and minus one for failure or to\n> return one for success and zero for failure. In the end I decided to go with\n> the former but I tend to agree with you that the latter would be easier to\n> understand.\n\nAgreed. V8 will switch to 0 for failure and 1 for success for the TEST,\nTEST_TODO, and check macros.\n\n\n> > > > +test_expect_success 'TAP output from unit tests' '\n> > > > [...]\n> > > > +\tok 19 - test with no checks returns -1\n> > > > +\t1..19\n> > > > +\tEOF\n> > > \n> > > Presumably t-basic will serve as a catalog of check_* functions and\n> > > the test binary, together with this test piece, will keep growing as\n> > > we gain features in the unit tests infrastructure. I wonder how\n> > > maintainable the above is, though. When we acquire new test, we\n> > > would need to renumber. What if multiple developers add new\n> > > features to the catalog at the same time?\n> \n> I think we could just add new tests to the end so we'd only need to change\n> the \"1..19\" line. That will become a source of merge conflicts if multiple\n> developers add new features at the same time though. Having several unit\n> test programs called from separate tests in t0080 might help with that.\n\nMy hope is that test-lib.c will not have to grow too extensively after\nthis series; that said, it's already been a pain to have to adjust the\nt0080 expected text several times just during development of this\nseries. I'll look into splitting this into several \"meta-tests\", but I'm\nnot sure I'll get to it for V8 yet.\n\n\n> > > > diff --git a/t/unit-tests/.gitignore b/t/unit-tests/.gitignore\n> > > > new file mode 100644\n> > > > index 0000000000..e292d58348\n> > > > --- /dev/null\n> > > > +++ b/t/unit-tests/.gitignore\n> > > > @@ -0,0 +1,2 @@\n> > > > +/t-basic\n> > > > +/t-strbuf\n> > > \n> > > Also, can we come up with some naming convention so that we do not\n> > > have to keep adding to this file every time we add a new test\n> > > script?\n> \n> Perhaps we should put the unit test binaries in a separate directory so we\n> can just add that directory to .gitignore.\n\nSounds good to me.\n"},{"id":"482883","messageId":"ZSQ6ZwoZDYtS5hmz@google.com","threadId":"59937","inReplyTo":"xmqq350hw6n7.fsf@gitster.g","subject":"Re: [PATCH v7 2/3] unit tests: add TAP unit test framework","fromName":"Josh Steadmon","fromEmail":"steadmon@google.com","sentAt":"2023-10-09T17:37:43Z","receivedAt":"2023-10-09T17:37:55Z","isPatch":true,"sender":{"key":"steadmon@google.com","avatar":"https://avatars.githubusercontent.com/u/2654920?v=4"},"body":"On 2023.08.17 17:12, Junio C Hamano wrote:\n> \n> What I am going to utter from here on are not complaints but purely\n> meant as questions. \n> \n> Would the resulting output and maintainability of the tests change\n> (improve, or worsen) if we introduce\n> \n> \tstatic void assert_empty_strbuf(struct strbuf *buf)\n> \t{\n> \t\tcheck_uint(buf->len, ==, 0);\n> check_uint(buf->alloc, ==, 0);\n> \t\tcheck(buf.buf == strbuf_slopbuf);\n> \t\tcheck_char(buf.buf[0], ==, '\\0');\n> \t}\n> \n> and call it from the setup() function to ensure that\n> strbuf_release(&buf) it calls after running customer test f() brings\n> the buffer in a reasonably initialized state? The t_static_init()\n> test should be able to say\n> \n> \tstatic void t_static_init(void)\n> \t{\n> \t\tstruct strbuf buf = STRBUF_INIT;\n> \t\tassert_empty_strbuf(&buf);\n> \t}\n> \n> if we did so, but is that a good thing or a bad thing (e.g. it may\n> make it harder to figure out where the real error came from, because\n> of the \"line number\" thing will not easily capture the caller of the\n> caller, perhaps)? \n\nI am unsure whether or not this is an improvement. While it would\ncertainly help readability and reduce duplication if this were\nproduction code, in test code it can often be more valuable to be\nverbose and explicit, so that individual broken test cases can be\nquickly understood without having to do a lot of cross referencing.\n\nI'll hold off on adding any more utility functions in t-strbuf for V8,\nbut if you or other folks feel strongly about it we can address it in\nV9.\n\n\n> > +\tcheck_char(buf.buf[0], ==, '\\0');\n> > +}\n> \n> > +static void t_dynamic_init(void)\n> > +{\n> > +\tstruct strbuf buf;\n> > +\n> > +\tstrbuf_init(&buf, 1024);\n> > +\tcheck_uint(buf.len, ==, 0);\n> > +\tcheck_uint(buf.alloc, >=, 1024);\n> > +\tcheck_char(buf.buf[0], ==, '\\0');\n> \n> Is it sensible to check buf.buf is not slopbuf at this point, or\n> does it make the test TOO intimate with the current implementation\n> detail?\n\nYes, I think this is too much of an internal detail. None of the users\nof strbuf ever reference it directly. Presumably for library-ish code,\nwe should stick to testing just the user-observable parts, not the\nimplementation.\n\n\n> > +\tcheck_char(buf->buf[0], ==, ch);\n> > +\tcheck_char(buf->buf[1], ==, '\\0');\n> > +}\n> \n> In any case, this t_addch() REQUIRES that incoming buf is empty,\n> doesn't it? I do not think it is sensible. I would have expected\n> that it would be more like\n> \n> \tt_addch(struct strbuf *buf, void *data)\n> \t{\n> \t\tchar ch = *(char *)data;\n> \t\tsize_t orig_alloc = buf->alloc;\n> \t\tsize_t orig_len = buf->len;\n> \n> \t\tif (!assert_sane_strbuf(buf))\n> \t\t\treturn;\n> strbuf_addch(buf, ch);\n> \t\tif (!assert_sane_strbuf(buf))\n> \t\t\treturn;\n> \t\tcheck_uint(buf->len, ==, orig_len + 1);\n> \t\tcheck_uint(buf->alloc, >=, orig_alloc);\n> check_char(buf->buf[buf->len - 1], ==, ch);\n> check_char(buf->buf[buf->len], ==, '\\0');\n> \t}\n> \n> to ensure that we can add a ch to a strbuf with any existing\n> contents and get a one-byte longer contents than before, with the\n> last byte of the buffer becoming 'ch' and still NUL terminated.\n> \n> And we protect ourselves with a helper that checks if the given\n> strbuf looks *sane*.\n\nYeah, in general I think this is a good improvement, but again I'm not\nsure if it's worth adding additional helpers. I'll try to rework this a\nbit in V8.\n\n\n> \tstatic int assert_sane_strbuf(struct strbuf *buf)\n> \t{\n> \t/* can use slopbuf only when the length is 0 */\n> \t\tif (buf->buf == strbuf_slopbuf)\n> \treturn (buf->len == 0);\n> \t\t/* everybody else must have non-NULL buffer */\n> \t\tif (buf->buf == NULL)\n> \t\t\treturn 0;\n> /* \n> \t\t * alloc must be at least 1 byte larger than len\n> \t\t * for the terminating NUL at the end.\n> \t\t */\n> \t\treturn ((buf->len + 1 <= buf->alloc) &&\n> \t\t \t(buf->buf[buf->len] == '\\0'));\n> \t}\n> \n> You can obviously use your check_foo() for the individual checks\n> done in this function to get a more detailed diagnosis, but because\n> I have confused myself enough by thinking about their polarity, I\n> wrote this in barebones comparison instead.\n> \n"},{"id":"482934","messageId":"cover.1696889529.git.steadmon@google.com","threadId":"59937","inReplyTo":"0169ce6fb9ccafc089b74ae406db0d1a8ff8ac65.1688165272.git.steadmon@google.com","subject":"[PATCH v8 0/3] Add unit test framework and project plan","fromName":"Josh Steadmon","fromEmail":"steadmon@google.com","sentAt":"2023-10-09T22:21:19Z","receivedAt":"2023-10-09T22:21:34Z","isPatch":true,"sender":{"key":"steadmon@google.com","avatar":"https://avatars.githubusercontent.com/u/2654920?v=4"},"body":"In our current testing environment, we spend a significant amount of\neffort crafting end-to-end tests for error conditions that could easily\nbe captured by unit tests (or we simply forgo some hard-to-setup and\nrare error conditions). Unit tests additionally provide stability to the\ncodebase and can simplify debugging through isolation. Turning parts of\nGit into libraries[1] gives us the ability to run unit tests on the\nlibraries and to write unit tests in C. Writing unit tests in pure C,\nrather than with our current shell/test-tool helper setup, simplifies\ntest setup, simplifies passing data around (no shell-isms required), and\nreduces testing runtime by not spawning a separate process for every\ntest invocation.\n\nThis series begins with a project document covering our goals for adding\nunit tests and a discussion of alternative frameworks considered, as\nwell as the features used to evaluate them. A rendered preview of this\ndoc can be found at [2]. It also adds Phillip Wood's TAP implemenation\n(with some slightly re-worked Makefile rules) and a sample strbuf unit\ntest. Finally, we modify the configs for GitHub and Cirrus CI to run the\nunit tests. Sample runs showing successful CI runs can be found at [3],\n[4], and [5].\n\n[1] https://lore.kernel.org/git/CAJoAoZ=Cig_kLocxKGax31sU7Xe4==BGzC__Bg2_pr7krNq6MA@mail.gmail.com/\n[2] https://github.com/steadmon/git/blob/unit-tests-asciidoc/Documentation/technical/unit-tests.adoc\n[3] https://github.com/steadmon/git/actions/runs/5884659246/job/15959781385#step:4:1803\n[4] https://github.com/steadmon/git/actions/runs/5884659246/job/15959938401#step:5:186\n[5] https://cirrus-ci.com/task/6126304366428160 (unrelated tests failed,\n but note that t-strbuf ran successfully)\n\nIn addition to reviewing the patches in this series, reviewers can help\nthis series progress by chiming in on these remaining TODOs:\n- Figure out if we should split t-basic.c into multiple meta-tests, to\n avoid merge conflicts and changes to expected text in\n t0080-unit-test-output.sh.\n- Figure out if we should de-duplicate assertions in t-strbuf.c at the\n cost of making tests less self-contained and diagnostic output less\n helpful.\n- Figure out if we should collect unit tests statistics similar to the\n \"counts\" files for shell tests\n- Decide if it's OK to wait on sharding unit tests across \"sliced\" CI\n instances\n- Provide guidelines for writing new unit tests\n\nChanges in v8:\n- Flipped return values for TEST, TEST_TODO, and check_* macros &\n functions. This makes it easier to reason about control flow for\n patterns like:\n if (check(some_condition)) { ... }\n- Moved unit test binaries to t/unit-tests/bin to simplify .gitignore\n patterns.\n- Removed testing of some strbuf implementation details in t-strbuf.c\n\n\nJosh Steadmon (2):\n unit tests: Add a project plan document\n ci: run unit tests in CI\n\nPhillip Wood (1):\n unit tests: add TAP unit test framework\n\n .cirrus.yml | 2 +-\n Documentation/Makefile | 1 +\n Documentation/technical/unit-tests.txt | 220 +++++++++++++++++\n Makefile | 28 ++-\n ci/run-build-and-tests.sh | 2 +\n ci/run-test-slice.sh | 5 +\n t/Makefile | 15 +-\n t/t0080-unit-test-output.sh | 58 +++++\n t/unit-tests/.gitignore | 1 +\n t/unit-tests/t-basic.c | 95 +++++++\n t/unit-tests/t-strbuf.c | 120 +++++++++\n t/unit-tests/test-lib.c | 329 +++++++++++++++++++++++++\n t/unit-tests/test-lib.h | 143 +++++++++++\n 13 files changed, 1014 insertions(+), 5 deletions(-)\n create mode 100644 Documentation/technical/unit-tests.txt\n create mode 100755 t/t0080-unit-test-output.sh\n create mode 100644 t/unit-tests/.gitignore\n create mode 100644 t/unit-tests/t-basic.c\n create mode 100644 t/unit-tests/t-strbuf.c\n create mode 100644 t/unit-tests/test-lib.c\n create mode 100644 t/unit-tests/test-lib.h\n\nRange-diff against v7:\n-: ---------- > 1: 81c5148a12 unit tests: Add a project plan document\n1: 3cc98d4045 ! 2: 00d3c95a81 unit tests: add TAP unit test framework\n @@ Commit message\n \n check_uint(buf.len, ==, 0);\n check_uint(buf.alloc, ==, 0);\n - if (check(buf.buf == strbuf_slopbuf))\n - return; /* avoid SIGSEV */\n check_char(buf.buf[0], ==, '\\0');\n }\n \n @@ Commit message\n checked by t0080-unit-test-output.sh. t-strbuf.c shows some example\n unit tests for strbuf.c\n \n - The unit tests can be built with \"make unit-tests\" (this works but the\n - Makefile changes need some further work). Once they have been built they\n - can be run manually (e.g t/unit-tests/t-strbuf) or with prove.\n + The unit tests will be built as part of the default \"make all\" target,\n + to avoid bitrot. If you wish to build just the unit tests, you can run\n + \"make build-unit-tests\". To run the tests, you can use \"make unit-tests\"\n + or run the test binaries directly, as in \"./t/unit-tests/bin/t-strbuf\".\n \n Signed-off-by: Phillip Wood \n @@ Makefile: TEST_BUILTINS_OBJS =\n THIRD_PARTY_SOURCES =\n +UNIT_TEST_PROGRAMS =\n +UNIT_TEST_DIR = t/unit-tests\n ++UNIT_TEST_BIN = $(UNIT_TEST_DIR)/bin\n \n # Having this variable in your environment would break pipelines because\n # you cause \"cd\" to echo its destination to stdout. It can also take\n @@ Makefile: THIRD_PARTY_SOURCES += compat/regex/%\n \n +UNIT_TEST_PROGRAMS += t-basic\n +UNIT_TEST_PROGRAMS += t-strbuf\n -+UNIT_TEST_PROGS = $(patsubst %,$(UNIT_TEST_DIR)/%$X,$(UNIT_TEST_PROGRAMS))\n ++UNIT_TEST_PROGS = $(patsubst %,$(UNIT_TEST_BIN)/%$X,$(UNIT_TEST_PROGRAMS))\n +UNIT_TEST_OBJS = $(patsubst %,$(UNIT_TEST_DIR)/%.o,$(UNIT_TEST_PROGRAMS))\n +UNIT_TEST_OBJS += $(UNIT_TEST_DIR)/test-lib.o\n +\n @@ Makefile: $(FUZZ_PROGRAMS): all\n \n fuzz-all: $(FUZZ_PROGRAMS)\n +\n -+$(UNIT_TEST_PROGS): $(UNIT_TEST_DIR)/%$X: $(UNIT_TEST_DIR)/%.o $(UNIT_TEST_DIR)/test-lib.o $(GITLIBS) GIT-LDFLAGS\n ++$(UNIT_TEST_BIN):\n ++\t@mkdir -p $(UNIT_TEST_BIN)\n ++\n ++$(UNIT_TEST_PROGS): $(UNIT_TEST_BIN)/%$X: $(UNIT_TEST_DIR)/%.o $(UNIT_TEST_DIR)/test-lib.o $(GITLIBS) GIT-LDFLAGS $(UNIT_TEST_BIN)\n +\t$(QUIET_LINK)$(CC) $(ALL_CFLAGS) -o $@ $(ALL_LDFLAGS) \\\n +\t\t$(filter %.o,$^) $(filter %.a,$^) $(LIBS)\n +\n @@ t/Makefile: TPERF = $(sort $(wildcard perf/p[0-9][0-9][0-9][0-9]-*.sh))\n TINTEROP = $(sort $(wildcard interop/i[0-9][0-9][0-9][0-9]-*.sh))\n CHAINLINTTESTS = $(sort $(patsubst chainlint/%.test,%,$(wildcard chainlint/*.test)))\n CHAINLINT = '$(PERL_PATH_SQ)' chainlint.pl\n -+UNIT_TESTS = $(sort $(filter-out %.h %.c %.o unit-tests/t-basic%,$(wildcard unit-tests/*)))\n ++UNIT_TESTS = $(sort $(filter-out unit-tests/bin/t-basic%,$(wildcard unit-tests/bin/t-*)))\n \n # `test-chainlint` (which is a dependency of `test-lint`, `test` and `prove`)\n # checks all tests in all scripts via a single invocation, so tell individual\n @@ t/t0080-unit-test-output.sh (new)\n +test_expect_success 'TAP output from unit tests' '\n +\tcat >expect <<-EOF &&\n +\tok 1 - passing test\n -+\tok 2 - passing test and assertion return 0\n ++\tok 2 - passing test and assertion return 1\n +\t# check \"1 == 2\" failed at t/unit-tests/t-basic.c:76\n +\t# left: 1\n +\t# right: 2\n +\tnot ok 3 - failing test\n -+\tok 4 - failing test and assertion return -1\n ++\tok 4 - failing test and assertion return 0\n +\tnot ok 5 - passing TEST_TODO() # TODO\n -+\tok 6 - passing TEST_TODO() returns 0\n ++\tok 6 - passing TEST_TODO() returns 1\n +\t# todo check ${SQ}check(x)${SQ} succeeded at t/unit-tests/t-basic.c:25\n +\tnot ok 7 - failing TEST_TODO()\n -+\tok 8 - failing TEST_TODO() returns -1\n ++\tok 8 - failing TEST_TODO() returns 0\n +\t# check \"0\" failed at t/unit-tests/t-basic.c:30\n +\t# skipping test - missing prerequisite\n +\t# skipping check ${SQ}1${SQ} at t/unit-tests/t-basic.c:32\n +\tok 9 - test_skip() # SKIP\n -+\tok 10 - skipped test returns 0\n ++\tok 10 - skipped test returns 1\n +\t# skipping test - missing prerequisite\n +\tok 11 - test_skip() inside TEST_TODO() # SKIP\n -+\tok 12 - test_skip() inside TEST_TODO() returns 0\n ++\tok 12 - test_skip() inside TEST_TODO() returns 1\n +\t# check \"0\" failed at t/unit-tests/t-basic.c:48\n +\tnot ok 13 - TEST_TODO() after failing check\n -+\tok 14 - TEST_TODO() after failing check returns -1\n ++\tok 14 - TEST_TODO() after failing check returns 0\n +\t# check \"0\" failed at t/unit-tests/t-basic.c:56\n +\tnot ok 15 - failing check after TEST_TODO()\n -+\tok 16 - failing check after TEST_TODO() returns -1\n ++\tok 16 - failing check after TEST_TODO() returns 0\n +\t# check \"!strcmp(\"\\thello\\\\\\\\\", \"there\\\"\\n\")\" failed at t/unit-tests/t-basic.c:61\n +\t# left: \"\\011hello\\\\\\\\\"\n +\t# right: \"there\\\"\\012\"\n @@ t/t0080-unit-test-output.sh (new)\n +\tnot ok 17 - messages from failing string and char comparison\n +\t# BUG: test has no checks at t/unit-tests/t-basic.c:91\n +\tnot ok 18 - test with no checks\n -+\tok 19 - test with no checks returns -1\n ++\tok 19 - test with no checks returns 0\n +\t1..19\n +\tEOF\n +\n -+\t! \"$GIT_BUILD_DIR\"/t/unit-tests/t-basic >actual &&\n ++\t! \"$GIT_BUILD_DIR\"/t/unit-tests/bin/t-basic >actual &&\n +\ttest_cmp expect actual\n +'\n +\n @@ t/t0080-unit-test-output.sh (new)\n \n ## t/unit-tests/.gitignore (new) ##\n @@\n -+/t-basic\n -+/t-strbuf\n ++/bin\n \n ## t/unit-tests/t-basic.c (new) ##\n @@\n @@ t/unit-tests/t-basic.c (new)\n +static int do_skip(void)\n +{\n +\ttest_skip(\"missing prerequisite\");\n -+\treturn 0;\n ++\treturn 1;\n +}\n +\n +static void t_skip_todo(void)\n @@ t/unit-tests/t-basic.c (new)\n +int cmd_main(int argc, const char **argv)\n +{\n +\ttest_res = TEST(check_res = check_int(1, ==, 1), \"passing test\");\n -+\tTEST(t_res(0), \"passing test and assertion return 0\");\n ++\tTEST(t_res(1), \"passing test and assertion return 1\");\n +\ttest_res = TEST(check_res = check_int(1, ==, 2), \"failing test\");\n -+\tTEST(t_res(-1), \"failing test and assertion return -1\");\n ++\tTEST(t_res(0), \"failing test and assertion return 0\");\n +\ttest_res = TEST(t_todo(0), \"passing TEST_TODO()\");\n -+\tTEST(t_res(0), \"passing TEST_TODO() returns 0\");\n ++\tTEST(t_res(1), \"passing TEST_TODO() returns 1\");\n +\ttest_res = TEST(t_todo(1), \"failing TEST_TODO()\");\n -+\tTEST(t_res(-1), \"failing TEST_TODO() returns -1\");\n ++\tTEST(t_res(0), \"failing TEST_TODO() returns 0\");\n +\ttest_res = TEST(t_skip(), \"test_skip()\");\n -+\tTEST(check_int(test_res, ==, 0), \"skipped test returns 0\");\n ++\tTEST(check_int(test_res, ==, 1), \"skipped test returns 1\");\n +\ttest_res = TEST(t_skip_todo(), \"test_skip() inside TEST_TODO()\");\n -+\tTEST(t_res(0), \"test_skip() inside TEST_TODO() returns 0\");\n ++\tTEST(t_res(1), \"test_skip() inside TEST_TODO() returns 1\");\n +\ttest_res = TEST(t_todo_after_fail(), \"TEST_TODO() after failing check\");\n -+\tTEST(check_int(test_res, ==, -1), \"TEST_TODO() after failing check returns -1\");\n ++\tTEST(check_int(test_res, ==, 0), \"TEST_TODO() after failing check returns 0\");\n +\ttest_res = TEST(t_fail_after_todo(), \"failing check after TEST_TODO()\");\n -+\tTEST(check_int(test_res, ==, -1), \"failing check after TEST_TODO() returns -1\");\n ++\tTEST(check_int(test_res, ==, 0), \"failing check after TEST_TODO() returns 0\");\n +\tTEST(t_messages(), \"messages from failing string and char comparison\");\n +\ttest_res = TEST(t_empty(), \"test with no checks\");\n -+\tTEST(check_int(test_res, ==, -1), \"test with no checks returns -1\");\n ++\tTEST(check_int(test_res, ==, 0), \"test with no checks returns 0\");\n +\n +\treturn test_done();\n +}\n @@ t/unit-tests/t-strbuf.c (new)\n +#include \"test-lib.h\"\n +#include \"strbuf.h\"\n +\n -+/* wrapper that supplies tests with an initialized strbuf */\n ++/* wrapper that supplies tests with an empty, initialized strbuf */\n +static void setup(void (*f)(struct strbuf*, void*), void *data)\n +{\n +\tstruct strbuf buf = STRBUF_INIT;\n @@ t/unit-tests/t-strbuf.c (new)\n +\tstrbuf_release(&buf);\n +\tcheck_uint(buf.len, ==, 0);\n +\tcheck_uint(buf.alloc, ==, 0);\n -+\tcheck(buf.buf == strbuf_slopbuf);\n -+\tcheck_char(buf.buf[0], ==, '\\0');\n ++}\n ++\n ++/* wrapper that supplies tests with a populated, initialized strbuf */\n ++static void setup_populated(void (*f)(struct strbuf*, void*), char *init_str, void *data)\n ++{\n ++\tstruct strbuf buf = STRBUF_INIT;\n ++\n ++\tstrbuf_addstr(&buf, init_str);\n ++\tcheck_uint(buf.len, ==, strlen(init_str));\n ++\tf(&buf, data);\n ++\tstrbuf_release(&buf);\n ++\tcheck_uint(buf.len, ==, 0);\n ++\tcheck_uint(buf.alloc, ==, 0);\n ++}\n ++\n ++static int assert_sane_strbuf(struct strbuf *buf)\n ++{\n ++\t/* Initialized strbufs should always have a non-NULL buffer */\n ++\tif (buf->buf == NULL)\n ++\t\treturn 0;\n ++\t/* Buffers should always be NUL-terminated */\n ++\tif (buf->buf[buf->len] != '\\0')\n ++\t\treturn 0;\n ++\t/*\n ++\t * Freshly-initialized strbufs may not have a dynamically allocated\n ++\t * buffer\n ++\t */\n ++\tif (buf->len == 0 && buf->alloc == 0)\n ++\t\treturn 1;\n ++\t/* alloc must be at least one byte larger than len */\n ++\treturn buf->len + 1 <= buf->alloc;\n +}\n +\n +static void t_static_init(void)\n @@ t/unit-tests/t-strbuf.c (new)\n +\n +\tcheck_uint(buf.len, ==, 0);\n +\tcheck_uint(buf.alloc, ==, 0);\n -+\tif (check(buf.buf == strbuf_slopbuf))\n -+\t\treturn; /* avoid de-referencing buf.buf */\n +\tcheck_char(buf.buf[0], ==, '\\0');\n +}\n +\n @@ t/unit-tests/t-strbuf.c (new)\n +\tstruct strbuf buf;\n +\n +\tstrbuf_init(&buf, 1024);\n ++\tcheck(assert_sane_strbuf(&buf));\n +\tcheck_uint(buf.len, ==, 0);\n +\tcheck_uint(buf.alloc, >=, 1024);\n +\tcheck_char(buf.buf[0], ==, '\\0');\n @@ t/unit-tests/t-strbuf.c (new)\n +{\n +\tconst char *p_ch = data;\n +\tconst char ch = *p_ch;\n ++\tsize_t orig_alloc = buf->alloc;\n ++\tsize_t orig_len = buf->len;\n +\n ++\tif (!check(assert_sane_strbuf(buf)))\n ++\t\treturn;\n +\tstrbuf_addch(buf, ch);\n -+\tif (check_uint(buf->len, ==, 1) ||\n -+\t check_uint(buf->alloc, >, 1))\n ++\tif (!check(assert_sane_strbuf(buf)))\n ++\t\treturn;\n ++\tif (!(check_uint(buf->len, ==, orig_len + 1) &&\n ++\t check_uint(buf->alloc, >=, orig_alloc)))\n +\t\treturn; /* avoid de-referencing buf->buf */\n -+\tcheck_char(buf->buf[0], ==, ch);\n -+\tcheck_char(buf->buf[1], ==, '\\0');\n ++\tcheck_char(buf->buf[buf->len - 1], ==, ch);\n ++\tcheck_char(buf->buf[buf->len], ==, '\\0');\n +}\n +\n +static void t_addstr(struct strbuf *buf, void *data)\n +{\n +\tconst char *text = data;\n +\tsize_t len = strlen(text);\n ++\tsize_t orig_alloc = buf->alloc;\n ++\tsize_t orig_len = buf->len;\n +\n ++\tif (!check(assert_sane_strbuf(buf)))\n ++\t\treturn;\n +\tstrbuf_addstr(buf, text);\n -+\tif (check_uint(buf->len, ==, len) ||\n -+\t check_uint(buf->alloc, >, len) ||\n -+\t check_char(buf->buf[len], ==, '\\0'))\n ++\tif (!check(assert_sane_strbuf(buf)))\n ++\t\treturn;\n ++\tif (!(check_uint(buf->len, ==, orig_len + len) &&\n ++\t check_uint(buf->alloc, >=, orig_alloc) &&\n ++\t check_uint(buf->alloc, >, orig_len + len) &&\n ++\t check_char(buf->buf[orig_len + len], ==, '\\0')))\n +\t return;\n -+\tcheck_str(buf->buf, text);\n ++\tcheck_str(buf->buf + orig_len, text);\n +}\n +\n +int cmd_main(int argc, const char **argv)\n +{\n -+\tif (TEST(t_static_init(), \"static initialization works\"))\n ++\tif (!TEST(t_static_init(), \"static initialization works\"))\n +\t\ttest_skip_all(\"STRBUF_INIT is broken\");\n +\tTEST(t_dynamic_init(), \"dynamic initialization works\");\n +\tTEST(setup(t_addch, \"a\"), \"strbuf_addch adds char\");\n +\tTEST(setup(t_addch, \"\"), \"strbuf_addch adds NUL char\");\n ++\tTEST(setup_populated(t_addch, \"initial value\", \"a\"),\n ++\t \"strbuf_addch appends to initial value\");\n +\tTEST(setup(t_addstr, \"hello there\"), \"strbuf_addstr adds string\");\n ++\tTEST(setup_populated(t_addstr, \"initial value\", \"hello there\"),\n ++\t \"strbuf_addstr appends string to initial value\");\n +\n +\treturn test_done();\n +}\n @@ t/unit-tests/test-lib.c (new)\n +\tva_end(ap);\n +\tctx.running = 0;\n +\tif (ctx.skip_all)\n -+\t\treturn 0;\n ++\t\treturn 1;\n +\tputc('\\n', stdout);\n +\tfflush(stdout);\n +\tctx.failed |= ctx.result == RESULT_FAILURE;\n +\n -+\treturn -(ctx.result == RESULT_FAILURE);\n ++\treturn ctx.result != RESULT_FAILURE;\n +}\n +\n +static void test_fail(void)\n @@ t/unit-tests/test-lib.c (new)\n +\n +\tif (ctx.result == RESULT_SKIP) {\n +\t\ttest_msg(\"skipping check '%s' at %s\", check, location);\n -+\t\treturn 0;\n ++\t\treturn 1;\n +\t} else if (!ctx.todo) {\n +\t\tif (ok) {\n +\t\t\ttest_pass();\n @@ t/unit-tests/test-lib.c (new)\n +\t\t}\n +\t}\n +\n -+\treturn -!ok;\n ++\treturn !!ok;\n +}\n +\n +void test__todo_begin(void)\n @@ t/unit-tests/test-lib.c (new)\n +\n +\tctx.todo = 0;\n +\tif (ctx.result == RESULT_SKIP)\n -+\t\treturn 0;\n -+\tif (!res) {\n ++\t\treturn 1;\n ++\tif (res) {\n +\t\ttest_msg(\"todo check '%s' succeeded at %s\", check, location);\n +\t\ttest_fail();\n +\t} else {\n +\t\ttest_todo();\n +\t}\n +\n -+\treturn -!res;\n ++\treturn !res;\n +}\n +\n +int check_bool_loc(const char *loc, const char *check, int ok)\n @@ t/unit-tests/test-lib.c (new)\n +{\n +\tint ret = test_assert(loc, check, ok);\n +\n -+\tif (ret) {\n ++\tif (!ret) {\n +\t\ttest_msg(\" left: %\"PRIdMAX, a);\n +\t\ttest_msg(\" right: %\"PRIdMAX, b);\n +\t}\n @@ t/unit-tests/test-lib.c (new)\n +{\n +\tint ret = test_assert(loc, check, ok);\n +\n -+\tif (ret) {\n ++\tif (!ret) {\n +\t\ttest_msg(\" left: %\"PRIuMAX, a);\n +\t\ttest_msg(\" right: %\"PRIuMAX, b);\n +\t}\n @@ t/unit-tests/test-lib.c (new)\n +{\n +\tint ret = test_assert(loc, check, ok);\n +\n -+\tif (ret) {\n ++\tif (!ret) {\n +\t\tfflush(stderr);\n +\t\tprint_char(\" left\", a);\n +\t\tprint_char(\" right\", b);\n @@ t/unit-tests/test-lib.c (new)\n +\tint ok = (!a && !b) || (a && b && !strcmp(a, b));\n +\tint ret = test_assert(loc, check, ok);\n +\n -+\tif (ret) {\n ++\tif (!ret) {\n +\t\tfflush(stderr);\n +\t\tprint_str(\" left\", a);\n +\t\tprint_str(\" right\", b);\n @@ t/unit-tests/test-lib.h (new)\n +#include \"git-compat-util.h\"\n +\n +/*\n -+ * Run a test function, returns 0 if the test succeeds, -1 if it\n ++ * Run a test function, returns 1 if the test succeeds, 0 if it\n + * fails. If test_skip_all() has been called then the test will not be\n + * run. The description for each test should be unique. For example:\n + *\n @@ t/unit-tests/test-lib.h (new)\n +void test_msg(const char *format, ...);\n +\n +/*\n -+ * Test checks are built around test_assert(). checks return 0 on\n -+ * success, -1 on failure. If any check fails then the test will\n ++ * Test checks are built around test_assert(). checks return 1 on\n ++ * success, 0 on failure. If any check fails then the test will\n + * fail. To create a custom check define a function that wraps\n + * test_assert() and a macro to wrap that function. For example:\n + *\n @@ t/unit-tests/test-lib.h (new)\n +\n +/*\n + * Wrap a check that is known to fail. If the check succeeds then the\n -+ * test will fail. Returns 0 if the check fails, -1 if it\n ++ * test will fail. Returns 1 if the check fails, 0 if it\n + * succeeds. For example:\n + *\n + * TEST_TODO(check(0));\n2: abf4dc41ac ! 3: aa1dfa4892 ci: run unit tests in CI\n @@ ci/run-test-slice.sh: group \"Run tests\" make --quiet -C t T=\"$(cd t &&\n +fi\n +\n check_unignored_build_artifacts\n -\n - ## t/Makefile ##\n -@@ t/Makefile: TPERF = $(sort $(wildcard perf/p[0-9][0-9][0-9][0-9]-*.sh))\n - TINTEROP = $(sort $(wildcard interop/i[0-9][0-9][0-9][0-9]-*.sh))\n - CHAINLINTTESTS = $(sort $(patsubst chainlint/%.test,%,$(wildcard chainlint/*.test)))\n - CHAINLINT = '$(PERL_PATH_SQ)' chainlint.pl\n --UNIT_TESTS = $(sort $(filter-out %.h %.c %.o unit-tests/t-basic%,$(wildcard unit-tests/*)))\n -+UNIT_TESTS = $(sort $(filter-out %.h %.c %.o unit-tests/t-basic%,$(wildcard unit-tests/t-*)))\n - \n - # `test-chainlint` (which is a dependency of `test-lint`, `test` and `prove`)\n - # checks all tests in all scripts via a single invocation, so tell individual\n\nbase-commit: a9e066fa63149291a55f383cfa113d8bdbdaa6b3\n-- \n2.42.0.609.gbb76f46606-goog\n\n"},{"id":"482935","messageId":"81c5148a1267b8f9ce432a950340f0fa16b4d773.1696889530.git.steadmon@google.com","threadId":"59937","inReplyTo":"cover.1696889529.git.steadmon@google.com","subject":"[PATCH v8 1/3] unit tests: Add a project plan document","fromName":"Josh Steadmon","fromEmail":"steadmon@google.com","sentAt":"2023-10-09T22:21:20Z","receivedAt":"2023-10-09T22:21:41Z","isPatch":true,"sender":{"key":"steadmon@google.com","avatar":"https://avatars.githubusercontent.com/u/2654920?v=4"},"body":"In our current testing environment, we spend a significant amount of\neffort crafting end-to-end tests for error conditions that could easily\nbe captured by unit tests (or we simply forgo some hard-to-setup and\nrare error conditions). Describe what we hope to accomplish by\nimplementing unit tests, and explain some open questions and milestones.\nDiscuss desired features for test frameworks/harnesses, and provide a\npreliminary comparison of several different frameworks.\n\nCo-authored-by: Calvin Wan \nSigned-off-by: Calvin Wan \nSigned-off-by: Josh Steadmon \n---\n Documentation/Makefile | 1 +\n Documentation/technical/unit-tests.txt | 220 +++++++++++++++++++++++++\n 2 files changed, 221 insertions(+)\n create mode 100644 Documentation/technical/unit-tests.txt\n\ndiff --git a/Documentation/Makefile b/Documentation/Makefile\nindex b629176d7d..3f2383a12c 100644\n--- a/Documentation/Makefile\n+++ b/Documentation/Makefile\n@@ -122,6 +122,7 @@ TECH_DOCS += technical/scalar\n TECH_DOCS += technical/send-pack-pipeline\n TECH_DOCS += technical/shallow\n TECH_DOCS += technical/trivial-merge\n+TECH_DOCS += technical/unit-tests\n SP_ARTICLES += $(TECH_DOCS)\n SP_ARTICLES += technical/api-index\n \ndiff --git a/Documentation/technical/unit-tests.txt b/Documentation/technical/unit-tests.txt\nnew file mode 100644\nindex 0000000000..b7a89cc838\n--- /dev/null\n+++ b/Documentation/technical/unit-tests.txt\n@@ -0,0 +1,220 @@\n+= Unit Testing\n+\n+In our current testing environment, we spend a significant amount of effort\n+crafting end-to-end tests for error conditions that could easily be captured by\n+unit tests (or we simply forgo some hard-to-setup and rare error conditions).\n+Unit tests additionally provide stability to the codebase and can simplify\n+debugging through isolation. Writing unit tests in pure C, rather than with our\n+current shell/test-tool helper setup, simplifies test setup, simplifies passing\n+data around (no shell-isms required), and reduces testing runtime by not\n+spawning a separate process for every test invocation.\n+\n+We believe that a large body of unit tests, living alongside the existing test\n+suite, will improve code quality for the Git project.\n+\n+== Definitions\n+\n+For the purposes of this document, we'll use *test framework* to refer to\n+projects that support writing test cases and running tests within the context\n+of a single executable. *Test harness* will refer to projects that manage\n+running multiple executables (each of which may contain multiple test cases) and\n+aggregating their results.\n+\n+In reality, these terms are not strictly defined, and many of the projects\n+discussed below contain features from both categories.\n+\n+For now, we will evaluate projects solely on their framework features. Since we\n+are relying on having TAP output (see below), we can assume that any framework\n+can be made to work with a harness that we can choose later.\n+\n+\n+== Choosing a framework\n+\n+We believe the best option is to implement a custom TAP framework for the Git\n+project. We use a version of the framework originally proposed in\n+https://lore.kernel.org/git/c902a166-98ce-afba-93f2-ea6027557176@gmail.com/[1].\n+\n+\n+== Choosing a test harness\n+\n+During upstream discussion, it was occasionally noted that `prove` provides many\n+convenient features, such as scheduling slower tests first, or re-running\n+previously failed tests.\n+\n+While we already support the use of `prove` as a test harness for the shell\n+tests, it is not strictly required. The t/Makefile allows running shell tests\n+directly (though with interleaved output if parallelism is enabled). Git\n+developers who wish to use `prove` as a more advanced harness can do so by\n+setting DEFAULT_TEST_TARGET=prove in their config.mak.\n+\n+We will follow a similar approach for unit tests: by default the test\n+executables will be run directly from the t/Makefile, but `prove` can be\n+configured with DEFAULT_UNIT_TEST_TARGET=prove.\n+\n+\n+== Framework selection\n+\n+There are a variety of features we can use to rank the candidate frameworks, and\n+those features have different priorities:\n+\n+* Critical features: we probably won't consider a framework without these\n+** Can we legally / easily use the project?\n+*** <>\n+*** <>\n+*** <>\n+*** <>\n+** Does the project support our bare-minimum needs?\n+*** <>\n+*** <>\n+*** <>\n+* Nice-to-have features:\n+** <>\n+** <>\n+** <>\n+* Tie-breaker stats\n+** <>\n+** <>\n+\n+[[license]]\n+=== License\n+\n+We must be able to legally use the framework in connection with Git. As Git is\n+licensed only under GPLv2, we must eliminate any LGPLv3, GPLv3, or Apache 2.0\n+projects.\n+\n+[[vendorable-or-ubiquitous]]\n+=== Vendorable or ubiquitous\n+\n+We want to avoid forcing Git developers to install new tools just to run unit\n+tests. Any prospective frameworks and harnesses must either be vendorable\n+(meaning, we can copy their source directly into Git's repository), or so\n+ubiquitous that it is reasonable to expect that most developers will have the\n+tools installed already.\n+\n+[[maintainable-extensible]]\n+=== Maintainable / extensible\n+\n+It is unlikely that any pre-existing project perfectly fits our needs, so any\n+project we select will need to be actively maintained and open to accepting\n+changes. Alternatively, assuming we are vendoring the source into our repo, it\n+must be simple enough that Git developers can feel comfortable making changes as\n+needed to our version.\n+\n+In the comparison table below, \"True\" means that the framework seems to have\n+active developers, that it is simple enough that Git developers can make changes\n+to it, and that the project seems open to accepting external contributions (or\n+that it is vendorable). \"Partial\" means that at least one of the above\n+conditions holds.\n+\n+[[major-platform-support]]\n+=== Major platform support\n+\n+At a bare minimum, unit-testing must work on Linux, MacOS, and Windows.\n+\n+In the comparison table below, \"True\" means that it works on all three major\n+platforms with no issues. \"Partial\" means that there may be annoyances on one or\n+more platforms, but it is still usable in principle.\n+\n+[[tap-support]]\n+=== TAP support\n+\n+The https://testanything.org/[Test Anything Protocol] is a text-based interface\n+that allows tests to communicate with a test harness. It is already used by\n+Git's integration test suite. Supporting TAP output is a mandatory feature for\n+any prospective test framework.\n+\n+In the comparison table below, \"True\" means this is natively supported.\n+\"Partial\" means TAP output must be generated by post-processing the native\n+output.\n+\n+Frameworks that do not have at least Partial support will not be evaluated\n+further.\n+\n+[[diagnostic-output]]\n+=== Diagnostic output\n+\n+When a test case fails, the framework must generate enough diagnostic output to\n+help developers find the appropriate test case in source code in order to debug\n+the failure.\n+\n+[[runtime-skippable-tests]]\n+=== Runtime-skippable tests\n+\n+Test authors may wish to skip certain test cases based on runtime circumstances,\n+so the framework should support this.\n+\n+[[parallel-execution]]\n+=== Parallel execution\n+\n+Ideally, we will build up a significant collection of unit test cases, most\n+likely split across multiple executables. It will be necessary to run these\n+tests in parallel to enable fast develop-test-debug cycles.\n+\n+In the comparison table below, \"True\" means that individual test cases within a\n+single test executable can be run in parallel. We assume that executable-level\n+parallelism can be handled by the test harness.\n+\n+[[mock-support]]\n+=== Mock support\n+\n+Unit test authors may wish to test code that interacts with objects that may be\n+inconvenient to handle in a test (e.g. interacting with a network service).\n+Mocking allows test authors to provide a fake implementation of these objects\n+for more convenient tests.\n+\n+[[signal-error-handling]]\n+=== Signal & error handling\n+\n+The test framework should fail gracefully when test cases are themselves buggy\n+or when they are interrupted by signals during runtime.\n+\n+[[project-kloc]]\n+=== Project KLOC\n+\n+The size of the project, in thousands of lines of code as measured by\n+https://dwheeler.com/sloccount/[sloccount] (rounded up to the next multiple of\n+1,000). As a tie-breaker, we probably prefer a project with fewer LOC.\n+\n+[[adoption]]\n+=== Adoption\n+\n+As a tie-breaker, we prefer a more widely-used project. We use the number of\n+GitHub / GitLab stars to estimate this.\n+\n+\n+=== Comparison\n+\n+[format=\"csv\",options=\"header\",width=\"33%\"]\n+|=====\n+Framework,\"<>\",\"<>\",\"<>\",\"<>\",\"<>\",\"<>\",\"<>\",\"<>\",\"<>\",\"<>\",\"<>\",\"<>\"\n+https://lore.kernel.org/git/c902a166-98ce-afba-93f2-ea6027557176@gmail.com/[Custom Git impl.],[lime-background]#GPL v2#,[lime-background]#True#,[lime-background]#True#,[lime-background]#True#,[lime-background]#True#,[lime-background]#True#,[lime-background]#True#,[red-background]#False#,[red-background]#False#,[red-background]#False#,1,0\n+https://github.com/silentbicycle/greatest[Greatest],[lime-background]#ISC#,[lime-background]#True#,[yellow-background]#Partial#,[lime-background]#True#,[yellow-background]#Partial#,[lime-background]#True#,[lime-background]#True#,[red-background]#False#,[red-background]#False#,[red-background]#False#,3,1400\n+https://github.com/Snaipe/Criterion[Criterion],[lime-background]#MIT#,[red-background]#False#,[yellow-background]#Partial#,[lime-background]#True#,[lime-background]#True#,[lime-background]#True#,[lime-background]#True#,[lime-background]#True#,[red-background]#False#,[lime-background]#True#,19,1800\n+https://github.com/rra/c-tap-harness/[C TAP],[lime-background]#Expat#,[lime-background]#True#,[yellow-background]#Partial#,[yellow-background]#Partial#,[lime-background]#True#,[red-background]#False#,[lime-background]#True#,[red-background]#False#,[red-background]#False#,[red-background]#False#,4,33\n+https://libcheck.github.io/check/[Check],[lime-background]#LGPL v2.1#,[red-background]#False#,[yellow-background]#Partial#,[lime-background]#True#,[lime-background]#True#,[lime-background]#True#,[red-background]#False#,[red-background]#False#,[red-background]#False#,[lime-background]#True#,17,973\n+|=====\n+\n+=== Additional framework candidates\n+\n+Several suggested frameworks have been eliminated from consideration:\n+\n+* Incompatible licenses:\n+** https://github.com/zorgnax/libtap[libtap] (LGPL v3)\n+** https://cmocka.org/[cmocka] (Apache 2.0)\n+* Missing source: https://www.kindahl.net/mytap/doc/index.html[MyTap]\n+* No TAP support:\n+** https://nemequ.github.io/munit/[µnit]\n+** https://github.com/google/cmockery[cmockery]\n+** https://github.com/lpabon/cmockery2[cmockery2]\n+** https://github.com/ThrowTheSwitch/Unity[Unity]\n+** https://github.com/siu/minunit[minunit]\n+** https://cunit.sourceforge.net/[CUnit]\n+\n+\n+== Milestones\n+\n+* Add useful tests of library-like code\n+* Integrate with\n+ https://lore.kernel.org/git/20230502211454.1673000-1-calvinwan@google.com/[stdlib\n+ work]\n+* Run alongside regular `make test` target\n-- \n2.42.0.609.gbb76f46606-goog\n\n"},{"id":"482937","messageId":"00d3c95a81449bf49c4ce992d862d7a858691840.1696889530.git.steadmon@google.com","threadId":"59937","inReplyTo":"cover.1696889529.git.steadmon@google.com","subject":"[PATCH v8 2/3] unit tests: add TAP unit test framework","fromName":"Josh Steadmon","fromEmail":"steadmon@google.com","sentAt":"2023-10-09T22:21:21Z","receivedAt":"2023-10-09T22:21:43Z","isPatch":true,"sender":{"key":"steadmon@google.com","avatar":"https://avatars.githubusercontent.com/u/2654920?v=4"},"body":"From: Phillip Wood \n\nThis patch contains an implementation for writing unit tests with TAP\noutput. Each test is a function that contains one or more checks. The\ntest is run with the TEST() macro and if any of the checks fail then the\ntest will fail. A complete program that tests STRBUF_INIT would look\nlike\n\n #include \"test-lib.h\"\n #include \"strbuf.h\"\n\n static void t_static_init(void)\n {\n struct strbuf buf = STRBUF_INIT;\n\n check_uint(buf.len, ==, 0);\n check_uint(buf.alloc, ==, 0);\n check_char(buf.buf[0], ==, '\\0');\n }\n\n int main(void)\n {\n TEST(t_static_init(), \"static initialization works);\n\n return test_done();\n }\n\nThe output of this program would be\n\n ok 1 - static initialization works\n 1..1\n\nIf any of the checks in a test fail then they print a diagnostic message\nto aid debugging and the test will be reported as failing. For example a\nfailing integer check would look like\n\n # check \"x >= 3\" failed at my-test.c:102\n # left: 2\n # right: 3\n not ok 1 - x is greater than or equal to three\n\nThere are a number of check functions implemented so far. check() checks\na boolean condition, check_int(), check_uint() and check_char() take two\nvalues to compare and a comparison operator. check_str() will check if\ntwo strings are equal. Custom checks are simple to implement as shown in\nthe comments above test_assert() in test-lib.h.\n\nTests can be skipped with test_skip() which can be supplied with a\nreason for skipping which it will print. Tests can print diagnostic\nmessages with test_msg(). Checks that are known to fail can be wrapped\nin TEST_TODO().\n\nThere are a couple of example test programs included in this\npatch. t-basic.c implements some self-tests and demonstrates the\ndiagnostic output for failing test. The output of this program is\nchecked by t0080-unit-test-output.sh. t-strbuf.c shows some example\nunit tests for strbuf.c\n\nThe unit tests will be built as part of the default \"make all\" target,\nto avoid bitrot. If you wish to build just the unit tests, you can run\n\"make build-unit-tests\". To run the tests, you can use \"make unit-tests\"\nor run the test binaries directly, as in \"./t/unit-tests/bin/t-strbuf\".\n\nSigned-off-by: Phillip Wood \nSigned-off-by: Josh Steadmon \n---\n Makefile | 28 ++-\n t/Makefile | 15 +-\n t/t0080-unit-test-output.sh | 58 +++++++\n t/unit-tests/.gitignore | 1 +\n t/unit-tests/t-basic.c | 95 +++++++++++\n t/unit-tests/t-strbuf.c | 120 +++++++++++++\n t/unit-tests/test-lib.c | 329 ++++++++++++++++++++++++++++++++++++\n t/unit-tests/test-lib.h | 143 ++++++++++++++++\n 8 files changed, 785 insertions(+), 4 deletions(-)\n create mode 100755 t/t0080-unit-test-output.sh\n create mode 100644 t/unit-tests/.gitignore\n create mode 100644 t/unit-tests/t-basic.c\n create mode 100644 t/unit-tests/t-strbuf.c\n create mode 100644 t/unit-tests/test-lib.c\n create mode 100644 t/unit-tests/test-lib.h\n\ndiff --git a/Makefile b/Makefile\nindex e440728c24..18c13f06c0 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -682,6 +682,9 @@ TEST_BUILTINS_OBJS =\n TEST_OBJS =\n TEST_PROGRAMS_NEED_X =\n THIRD_PARTY_SOURCES =\n+UNIT_TEST_PROGRAMS =\n+UNIT_TEST_DIR = t/unit-tests\n+UNIT_TEST_BIN = $(UNIT_TEST_DIR)/bin\n \n # Having this variable in your environment would break pipelines because\n # you cause \"cd\" to echo its destination to stdout. It can also take\n@@ -1331,6 +1334,12 @@ THIRD_PARTY_SOURCES += compat/regex/%\n THIRD_PARTY_SOURCES += sha1collisiondetection/%\n THIRD_PARTY_SOURCES += sha1dc/%\n \n+UNIT_TEST_PROGRAMS += t-basic\n+UNIT_TEST_PROGRAMS += t-strbuf\n+UNIT_TEST_PROGS = $(patsubst %,$(UNIT_TEST_BIN)/%$X,$(UNIT_TEST_PROGRAMS))\n+UNIT_TEST_OBJS = $(patsubst %,$(UNIT_TEST_DIR)/%.o,$(UNIT_TEST_PROGRAMS))\n+UNIT_TEST_OBJS += $(UNIT_TEST_DIR)/test-lib.o\n+\n # xdiff and reftable libs may in turn depend on what is in libgit.a\n GITLIBS = common-main.o $(LIB_FILE) $(XDIFF_LIB) $(REFTABLE_LIB) $(LIB_FILE)\n EXTLIBS =\n@@ -2672,6 +2681,7 @@ OBJECTS += $(TEST_OBJS)\n OBJECTS += $(XDIFF_OBJS)\n OBJECTS += $(FUZZ_OBJS)\n OBJECTS += $(REFTABLE_OBJS) $(REFTABLE_TEST_OBJS)\n+OBJECTS += $(UNIT_TEST_OBJS)\n \n ifndef NO_CURL\n \tOBJECTS += http.o http-walker.o remote-curl.o\n@@ -3167,7 +3177,7 @@ endif\n \n test_bindir_programs := $(patsubst %,bin-wrappers/%,$(BINDIR_PROGRAMS_NEED_X) $(BINDIR_PROGRAMS_NO_X) $(TEST_PROGRAMS_NEED_X))\n \n-all:: $(TEST_PROGRAMS) $(test_bindir_programs)\n+all:: $(TEST_PROGRAMS) $(test_bindir_programs) $(UNIT_TEST_PROGS)\n \n bin-wrappers/%: wrap-for-bin.sh\n \t$(call mkdir_p_parent_template)\n@@ -3592,7 +3602,7 @@ endif\n \n artifacts-tar:: $(ALL_COMMANDS_TO_INSTALL) $(SCRIPT_LIB) $(OTHER_PROGRAMS) \\\n \t\tGIT-BUILD-OPTIONS $(TEST_PROGRAMS) $(test_bindir_programs) \\\n-\t\t$(MOFILES)\n+\t\t$(UNIT_TEST_PROGS) $(MOFILES)\n \t$(QUIET_SUBDIR0)templates $(QUIET_SUBDIR1) \\\n \t\tSHELL_PATH='$(SHELL_PATH_SQ)' PERL_PATH='$(PERL_PATH_SQ)'\n \ttest -n \"$(ARTIFACTS_DIRECTORY)\"\n@@ -3653,7 +3663,7 @@ clean: profile-clean coverage-clean cocciclean\n \t$(RM) $(OBJECTS)\n \t$(RM) $(LIB_FILE) $(XDIFF_LIB) $(REFTABLE_LIB) $(REFTABLE_TEST_LIB)\n \t$(RM) $(ALL_PROGRAMS) $(SCRIPT_LIB) $(BUILT_INS) $(OTHER_PROGRAMS)\n-\t$(RM) $(TEST_PROGRAMS)\n+\t$(RM) $(TEST_PROGRAMS) $(UNIT_TEST_PROGS)\n \t$(RM) $(FUZZ_PROGRAMS)\n \t$(RM) $(SP_OBJ)\n \t$(RM) $(HCC)\n@@ -3831,3 +3841,15 @@ $(FUZZ_PROGRAMS): all\n \t\t$(XDIFF_OBJS) $(EXTLIBS) git.o $@.o $(LIB_FUZZING_ENGINE) -o $@\n \n fuzz-all: $(FUZZ_PROGRAMS)\n+\n+$(UNIT_TEST_BIN):\n+\t@mkdir -p $(UNIT_TEST_BIN)\n+\n+$(UNIT_TEST_PROGS): $(UNIT_TEST_BIN)/%$X: $(UNIT_TEST_DIR)/%.o $(UNIT_TEST_DIR)/test-lib.o $(GITLIBS) GIT-LDFLAGS $(UNIT_TEST_BIN)\n+\t$(QUIET_LINK)$(CC) $(ALL_CFLAGS) -o $@ $(ALL_LDFLAGS) \\\n+\t\t$(filter %.o,$^) $(filter %.a,$^) $(LIBS)\n+\n+.PHONY: build-unit-tests unit-tests\n+build-unit-tests: $(UNIT_TEST_PROGS)\n+unit-tests: $(UNIT_TEST_PROGS)\n+\t$(MAKE) -C t/ unit-tests\ndiff --git a/t/Makefile b/t/Makefile\nindex 3e00cdd801..75d9330437 100644\n--- a/t/Makefile\n+++ b/t/Makefile\n@@ -17,6 +17,7 @@ TAR ?= $(TAR)\n RM ?= rm -f\n PROVE ?= prove\n DEFAULT_TEST_TARGET ?= test\n+DEFAULT_UNIT_TEST_TARGET ?= unit-tests-raw\n TEST_LINT ?= test-lint\n \n ifdef TEST_OUTPUT_DIRECTORY\n@@ -41,6 +42,7 @@ TPERF = $(sort $(wildcard perf/p[0-9][0-9][0-9][0-9]-*.sh))\n TINTEROP = $(sort $(wildcard interop/i[0-9][0-9][0-9][0-9]-*.sh))\n CHAINLINTTESTS = $(sort $(patsubst chainlint/%.test,%,$(wildcard chainlint/*.test)))\n CHAINLINT = '$(PERL_PATH_SQ)' chainlint.pl\n+UNIT_TESTS = $(sort $(filter-out unit-tests/bin/t-basic%,$(wildcard unit-tests/bin/t-*)))\n \n # `test-chainlint` (which is a dependency of `test-lint`, `test` and `prove`)\n # checks all tests in all scripts via a single invocation, so tell individual\n@@ -65,6 +67,17 @@ prove: pre-clean check-chainlint $(TEST_LINT)\n $(T):\n \t@echo \"*** $@ ***\"; '$(TEST_SHELL_PATH_SQ)' $@ $(GIT_TEST_OPTS)\n \n+$(UNIT_TESTS):\n+\t@echo \"*** $@ ***\"; $@\n+\n+.PHONY: unit-tests unit-tests-raw unit-tests-prove\n+unit-tests: $(DEFAULT_UNIT_TEST_TARGET)\n+\n+unit-tests-raw: $(UNIT_TESTS)\n+\n+unit-tests-prove:\n+\t@echo \"*** prove - unit tests ***\"; $(PROVE) $(GIT_PROVE_OPTS) $(UNIT_TESTS)\n+\n pre-clean:\n \t$(RM) -r '$(TEST_RESULTS_DIRECTORY_SQ)'\n \n@@ -149,4 +162,4 @@ perf:\n \t$(MAKE) -C perf/ all\n \n .PHONY: pre-clean $(T) aggregate-results clean valgrind perf \\\n-\tcheck-chainlint clean-chainlint test-chainlint\n+\tcheck-chainlint clean-chainlint test-chainlint $(UNIT_TESTS)\ndiff --git a/t/t0080-unit-test-output.sh b/t/t0080-unit-test-output.sh\nnew file mode 100755\nindex 0000000000..961b54b06c\n--- /dev/null\n+++ b/t/t0080-unit-test-output.sh\n@@ -0,0 +1,58 @@\n+#!/bin/sh\n+\n+test_description='Test the output of the unit test framework'\n+\n+. ./test-lib.sh\n+\n+test_expect_success 'TAP output from unit tests' '\n+\tcat >expect <<-EOF &&\n+\tok 1 - passing test\n+\tok 2 - passing test and assertion return 1\n+\t# check \"1 == 2\" failed at t/unit-tests/t-basic.c:76\n+\t# left: 1\n+\t# right: 2\n+\tnot ok 3 - failing test\n+\tok 4 - failing test and assertion return 0\n+\tnot ok 5 - passing TEST_TODO() # TODO\n+\tok 6 - passing TEST_TODO() returns 1\n+\t# todo check ${SQ}check(x)${SQ} succeeded at t/unit-tests/t-basic.c:25\n+\tnot ok 7 - failing TEST_TODO()\n+\tok 8 - failing TEST_TODO() returns 0\n+\t# check \"0\" failed at t/unit-tests/t-basic.c:30\n+\t# skipping test - missing prerequisite\n+\t# skipping check ${SQ}1${SQ} at t/unit-tests/t-basic.c:32\n+\tok 9 - test_skip() # SKIP\n+\tok 10 - skipped test returns 1\n+\t# skipping test - missing prerequisite\n+\tok 11 - test_skip() inside TEST_TODO() # SKIP\n+\tok 12 - test_skip() inside TEST_TODO() returns 1\n+\t# check \"0\" failed at t/unit-tests/t-basic.c:48\n+\tnot ok 13 - TEST_TODO() after failing check\n+\tok 14 - TEST_TODO() after failing check returns 0\n+\t# check \"0\" failed at t/unit-tests/t-basic.c:56\n+\tnot ok 15 - failing check after TEST_TODO()\n+\tok 16 - failing check after TEST_TODO() returns 0\n+\t# check \"!strcmp(\"\\thello\\\\\\\\\", \"there\\\"\\n\")\" failed at t/unit-tests/t-basic.c:61\n+\t# left: \"\\011hello\\\\\\\\\"\n+\t# right: \"there\\\"\\012\"\n+\t# check \"!strcmp(\"NULL\", NULL)\" failed at t/unit-tests/t-basic.c:62\n+\t# left: \"NULL\"\n+\t# right: NULL\n+\t# check \"${SQ}a${SQ} == ${SQ}\\n${SQ}\" failed at t/unit-tests/t-basic.c:63\n+\t# left: ${SQ}a${SQ}\n+\t# right: ${SQ}\\012${SQ}\n+\t# check \"${SQ}\\\\\\\\${SQ} == ${SQ}\\\\${SQ}${SQ}\" failed at t/unit-tests/t-basic.c:64\n+\t# left: ${SQ}\\\\\\\\${SQ}\n+\t# right: ${SQ}\\\\${SQ}${SQ}\n+\tnot ok 17 - messages from failing string and char comparison\n+\t# BUG: test has no checks at t/unit-tests/t-basic.c:91\n+\tnot ok 18 - test with no checks\n+\tok 19 - test with no checks returns 0\n+\t1..19\n+\tEOF\n+\n+\t! \"$GIT_BUILD_DIR\"/t/unit-tests/bin/t-basic >actual &&\n+\ttest_cmp expect actual\n+'\n+\n+test_done\ndiff --git a/t/unit-tests/.gitignore b/t/unit-tests/.gitignore\nnew file mode 100644\nindex 0000000000..5e56e040ec\n--- /dev/null\n+++ b/t/unit-tests/.gitignore\n@@ -0,0 +1 @@\n+/bin\ndiff --git a/t/unit-tests/t-basic.c b/t/unit-tests/t-basic.c\nnew file mode 100644\nindex 0000000000..fda1ae59a6\n--- /dev/null\n+++ b/t/unit-tests/t-basic.c\n@@ -0,0 +1,95 @@\n+#include \"test-lib.h\"\n+\n+/*\n+ * The purpose of this \"unit test\" is to verify a few invariants of the unit\n+ * test framework itself, as well as to provide examples of output from actually\n+ * failing tests. As such, it is intended that this test fails, and thus it\n+ * should not be run as part of `make unit-tests`. Instead, we verify it behaves\n+ * as expected in the integration test t0080-unit-test-output.sh\n+ */\n+\n+/* Used to store the return value of check_int(). */\n+static int check_res;\n+\n+/* Used to store the return value of TEST(). */\n+static int test_res;\n+\n+static void t_res(int expect)\n+{\n+\tcheck_int(check_res, ==, expect);\n+\tcheck_int(test_res, ==, expect);\n+}\n+\n+static void t_todo(int x)\n+{\n+\tcheck_res = TEST_TODO(check(x));\n+}\n+\n+static void t_skip(void)\n+{\n+\tcheck(0);\n+\ttest_skip(\"missing prerequisite\");\n+\tcheck(1);\n+}\n+\n+static int do_skip(void)\n+{\n+\ttest_skip(\"missing prerequisite\");\n+\treturn 1;\n+}\n+\n+static void t_skip_todo(void)\n+{\n+\tcheck_res = TEST_TODO(do_skip());\n+}\n+\n+static void t_todo_after_fail(void)\n+{\n+\tcheck(0);\n+\tTEST_TODO(check(0));\n+}\n+\n+static void t_fail_after_todo(void)\n+{\n+\tcheck(1);\n+\tTEST_TODO(check(0));\n+\tcheck(0);\n+}\n+\n+static void t_messages(void)\n+{\n+\tcheck_str(\"\\thello\\\\\", \"there\\\"\\n\");\n+\tcheck_str(\"NULL\", NULL);\n+\tcheck_char('a', ==, '\\n');\n+\tcheck_char('\\\\', ==, '\\'');\n+}\n+\n+static void t_empty(void)\n+{\n+\t; /* empty */\n+}\n+\n+int cmd_main(int argc, const char **argv)\n+{\n+\ttest_res = TEST(check_res = check_int(1, ==, 1), \"passing test\");\n+\tTEST(t_res(1), \"passing test and assertion return 1\");\n+\ttest_res = TEST(check_res = check_int(1, ==, 2), \"failing test\");\n+\tTEST(t_res(0), \"failing test and assertion return 0\");\n+\ttest_res = TEST(t_todo(0), \"passing TEST_TODO()\");\n+\tTEST(t_res(1), \"passing TEST_TODO() returns 1\");\n+\ttest_res = TEST(t_todo(1), \"failing TEST_TODO()\");\n+\tTEST(t_res(0), \"failing TEST_TODO() returns 0\");\n+\ttest_res = TEST(t_skip(), \"test_skip()\");\n+\tTEST(check_int(test_res, ==, 1), \"skipped test returns 1\");\n+\ttest_res = TEST(t_skip_todo(), \"test_skip() inside TEST_TODO()\");\n+\tTEST(t_res(1), \"test_skip() inside TEST_TODO() returns 1\");\n+\ttest_res = TEST(t_todo_after_fail(), \"TEST_TODO() after failing check\");\n+\tTEST(check_int(test_res, ==, 0), \"TEST_TODO() after failing check returns 0\");\n+\ttest_res = TEST(t_fail_after_todo(), \"failing check after TEST_TODO()\");\n+\tTEST(check_int(test_res, ==, 0), \"failing check after TEST_TODO() returns 0\");\n+\tTEST(t_messages(), \"messages from failing string and char comparison\");\n+\ttest_res = TEST(t_empty(), \"test with no checks\");\n+\tTEST(check_int(test_res, ==, 0), \"test with no checks returns 0\");\n+\n+\treturn test_done();\n+}\ndiff --git a/t/unit-tests/t-strbuf.c b/t/unit-tests/t-strbuf.c\nnew file mode 100644\nindex 0000000000..c2fcb0cbd6\n--- /dev/null\n+++ b/t/unit-tests/t-strbuf.c\n@@ -0,0 +1,120 @@\n+#include \"test-lib.h\"\n+#include \"strbuf.h\"\n+\n+/* wrapper that supplies tests with an empty, initialized strbuf */\n+static void setup(void (*f)(struct strbuf*, void*), void *data)\n+{\n+\tstruct strbuf buf = STRBUF_INIT;\n+\n+\tf(&buf, data);\n+\tstrbuf_release(&buf);\n+\tcheck_uint(buf.len, ==, 0);\n+\tcheck_uint(buf.alloc, ==, 0);\n+}\n+\n+/* wrapper that supplies tests with a populated, initialized strbuf */\n+static void setup_populated(void (*f)(struct strbuf*, void*), char *init_str, void *data)\n+{\n+\tstruct strbuf buf = STRBUF_INIT;\n+\n+\tstrbuf_addstr(&buf, init_str);\n+\tcheck_uint(buf.len, ==, strlen(init_str));\n+\tf(&buf, data);\n+\tstrbuf_release(&buf);\n+\tcheck_uint(buf.len, ==, 0);\n+\tcheck_uint(buf.alloc, ==, 0);\n+}\n+\n+static int assert_sane_strbuf(struct strbuf *buf)\n+{\n+\t/* Initialized strbufs should always have a non-NULL buffer */\n+\tif (buf->buf == NULL)\n+\t\treturn 0;\n+\t/* Buffers should always be NUL-terminated */\n+\tif (buf->buf[buf->len] != '\\0')\n+\t\treturn 0;\n+\t/*\n+\t * Freshly-initialized strbufs may not have a dynamically allocated\n+\t * buffer\n+\t */\n+\tif (buf->len == 0 && buf->alloc == 0)\n+\t\treturn 1;\n+\t/* alloc must be at least one byte larger than len */\n+\treturn buf->len + 1 <= buf->alloc;\n+}\n+\n+static void t_static_init(void)\n+{\n+\tstruct strbuf buf = STRBUF_INIT;\n+\n+\tcheck_uint(buf.len, ==, 0);\n+\tcheck_uint(buf.alloc, ==, 0);\n+\tcheck_char(buf.buf[0], ==, '\\0');\n+}\n+\n+static void t_dynamic_init(void)\n+{\n+\tstruct strbuf buf;\n+\n+\tstrbuf_init(&buf, 1024);\n+\tcheck(assert_sane_strbuf(&buf));\n+\tcheck_uint(buf.len, ==, 0);\n+\tcheck_uint(buf.alloc, >=, 1024);\n+\tcheck_char(buf.buf[0], ==, '\\0');\n+\tstrbuf_release(&buf);\n+}\n+\n+static void t_addch(struct strbuf *buf, void *data)\n+{\n+\tconst char *p_ch = data;\n+\tconst char ch = *p_ch;\n+\tsize_t orig_alloc = buf->alloc;\n+\tsize_t orig_len = buf->len;\n+\n+\tif (!check(assert_sane_strbuf(buf)))\n+\t\treturn;\n+\tstrbuf_addch(buf, ch);\n+\tif (!check(assert_sane_strbuf(buf)))\n+\t\treturn;\n+\tif (!(check_uint(buf->len, ==, orig_len + 1) &&\n+\t check_uint(buf->alloc, >=, orig_alloc)))\n+\t\treturn; /* avoid de-referencing buf->buf */\n+\tcheck_char(buf->buf[buf->len - 1], ==, ch);\n+\tcheck_char(buf->buf[buf->len], ==, '\\0');\n+}\n+\n+static void t_addstr(struct strbuf *buf, void *data)\n+{\n+\tconst char *text = data;\n+\tsize_t len = strlen(text);\n+\tsize_t orig_alloc = buf->alloc;\n+\tsize_t orig_len = buf->len;\n+\n+\tif (!check(assert_sane_strbuf(buf)))\n+\t\treturn;\n+\tstrbuf_addstr(buf, text);\n+\tif (!check(assert_sane_strbuf(buf)))\n+\t\treturn;\n+\tif (!(check_uint(buf->len, ==, orig_len + len) &&\n+\t check_uint(buf->alloc, >=, orig_alloc) &&\n+\t check_uint(buf->alloc, >, orig_len + len) &&\n+\t check_char(buf->buf[orig_len + len], ==, '\\0')))\n+\t return;\n+\tcheck_str(buf->buf + orig_len, text);\n+}\n+\n+int cmd_main(int argc, const char **argv)\n+{\n+\tif (!TEST(t_static_init(), \"static initialization works\"))\n+\t\ttest_skip_all(\"STRBUF_INIT is broken\");\n+\tTEST(t_dynamic_init(), \"dynamic initialization works\");\n+\tTEST(setup(t_addch, \"a\"), \"strbuf_addch adds char\");\n+\tTEST(setup(t_addch, \"\"), \"strbuf_addch adds NUL char\");\n+\tTEST(setup_populated(t_addch, \"initial value\", \"a\"),\n+\t \"strbuf_addch appends to initial value\");\n+\tTEST(setup(t_addstr, \"hello there\"), \"strbuf_addstr adds string\");\n+\tTEST(setup_populated(t_addstr, \"initial value\", \"hello there\"),\n+\t \"strbuf_addstr appends string to initial value\");\n+\n+\treturn test_done();\n+}\ndiff --git a/t/unit-tests/test-lib.c b/t/unit-tests/test-lib.c\nnew file mode 100644\nindex 0000000000..b20f543121\n--- /dev/null\n+++ b/t/unit-tests/test-lib.c\n@@ -0,0 +1,329 @@\n+#include \"test-lib.h\"\n+\n+enum result {\n+\tRESULT_NONE,\n+\tRESULT_FAILURE,\n+\tRESULT_SKIP,\n+\tRESULT_SUCCESS,\n+\tRESULT_TODO\n+};\n+\n+static struct {\n+\tenum result result;\n+\tint count;\n+\tunsigned failed :1;\n+\tunsigned lazy_plan :1;\n+\tunsigned running :1;\n+\tunsigned skip_all :1;\n+\tunsigned todo :1;\n+} ctx = {\n+\t.lazy_plan = 1,\n+\t.result = RESULT_NONE,\n+};\n+\n+static void msg_with_prefix(const char *prefix, const char *format, va_list ap)\n+{\n+\tfflush(stderr);\n+\tif (prefix)\n+\t\tfprintf(stdout, \"%s\", prefix);\n+\tvprintf(format, ap); /* TODO: handle newlines */\n+\tputc('\\n', stdout);\n+\tfflush(stdout);\n+}\n+\n+void test_msg(const char *format, ...)\n+{\n+\tva_list ap;\n+\n+\tva_start(ap, format);\n+\tmsg_with_prefix(\"# \", format, ap);\n+\tva_end(ap);\n+}\n+\n+void test_plan(int count)\n+{\n+\tassert(!ctx.running);\n+\n+\tfflush(stderr);\n+\tprintf(\"1..%d\\n\", count);\n+\tfflush(stdout);\n+\tctx.lazy_plan = 0;\n+}\n+\n+int test_done(void)\n+{\n+\tassert(!ctx.running);\n+\n+\tif (ctx.lazy_plan)\n+\t\ttest_plan(ctx.count);\n+\n+\treturn ctx.failed;\n+}\n+\n+void test_skip(const char *format, ...)\n+{\n+\tva_list ap;\n+\n+\tassert(ctx.running);\n+\n+\tctx.result = RESULT_SKIP;\n+\tva_start(ap, format);\n+\tif (format)\n+\t\tmsg_with_prefix(\"# skipping test - \", format, ap);\n+\tva_end(ap);\n+}\n+\n+void test_skip_all(const char *format, ...)\n+{\n+\tva_list ap;\n+\tconst char *prefix;\n+\n+\tif (!ctx.count && ctx.lazy_plan) {\n+\t\t/* We have not printed a test plan yet */\n+\t\tprefix = \"1..0 # SKIP \";\n+\t\tctx.lazy_plan = 0;\n+\t} else {\n+\t\t/* We have already printed a test plan */\n+\t\tprefix = \"Bail out! # \";\n+\t\tctx.failed = 1;\n+\t}\n+\tctx.skip_all = 1;\n+\tctx.result = RESULT_SKIP;\n+\tva_start(ap, format);\n+\tmsg_with_prefix(prefix, format, ap);\n+\tva_end(ap);\n+}\n+\n+int test__run_begin(void)\n+{\n+\tassert(!ctx.running);\n+\n+\tctx.count++;\n+\tctx.result = RESULT_NONE;\n+\tctx.running = 1;\n+\n+\treturn ctx.skip_all;\n+}\n+\n+static void print_description(const char *format, va_list ap)\n+{\n+\tif (format) {\n+\t\tfputs(\" - \", stdout);\n+\t\tvprintf(format, ap);\n+\t}\n+}\n+\n+int test__run_end(int was_run UNUSED, const char *location, const char *format, ...)\n+{\n+\tva_list ap;\n+\n+\tassert(ctx.running);\n+\tassert(!ctx.todo);\n+\n+\tfflush(stderr);\n+\tva_start(ap, format);\n+\tif (!ctx.skip_all) {\n+\t\tswitch (ctx.result) {\n+\t\tcase RESULT_SUCCESS:\n+\t\t\tprintf(\"ok %d\", ctx.count);\n+\t\t\tprint_description(format, ap);\n+\t\t\tbreak;\n+\n+\t\tcase RESULT_FAILURE:\n+\t\t\tprintf(\"not ok %d\", ctx.count);\n+\t\t\tprint_description(format, ap);\n+\t\t\tbreak;\n+\n+\t\tcase RESULT_TODO:\n+\t\t\tprintf(\"not ok %d\", ctx.count);\n+\t\t\tprint_description(format, ap);\n+\t\t\tprintf(\" # TODO\");\n+\t\t\tbreak;\n+\n+\t\tcase RESULT_SKIP:\n+\t\t\tprintf(\"ok %d\", ctx.count);\n+\t\t\tprint_description(format, ap);\n+\t\t\tprintf(\" # SKIP\");\n+\t\t\tbreak;\n+\n+\t\tcase RESULT_NONE:\n+\t\t\ttest_msg(\"BUG: test has no checks at %s\", location);\n+\t\t\tprintf(\"not ok %d\", ctx.count);\n+\t\t\tprint_description(format, ap);\n+\t\t\tctx.result = RESULT_FAILURE;\n+\t\t\tbreak;\n+\t\t}\n+\t}\n+\tva_end(ap);\n+\tctx.running = 0;\n+\tif (ctx.skip_all)\n+\t\treturn 1;\n+\tputc('\\n', stdout);\n+\tfflush(stdout);\n+\tctx.failed |= ctx.result == RESULT_FAILURE;\n+\n+\treturn ctx.result != RESULT_FAILURE;\n+}\n+\n+static void test_fail(void)\n+{\n+\tassert(ctx.result != RESULT_SKIP);\n+\n+\tctx.result = RESULT_FAILURE;\n+}\n+\n+static void test_pass(void)\n+{\n+\tassert(ctx.result != RESULT_SKIP);\n+\n+\tif (ctx.result == RESULT_NONE)\n+\t\tctx.result = RESULT_SUCCESS;\n+}\n+\n+static void test_todo(void)\n+{\n+\tassert(ctx.result != RESULT_SKIP);\n+\n+\tif (ctx.result != RESULT_FAILURE)\n+\t\tctx.result = RESULT_TODO;\n+}\n+\n+int test_assert(const char *location, const char *check, int ok)\n+{\n+\tassert(ctx.running);\n+\n+\tif (ctx.result == RESULT_SKIP) {\n+\t\ttest_msg(\"skipping check '%s' at %s\", check, location);\n+\t\treturn 1;\n+\t} else if (!ctx.todo) {\n+\t\tif (ok) {\n+\t\t\ttest_pass();\n+\t\t} else {\n+\t\t\ttest_msg(\"check \\\"%s\\\" failed at %s\", check, location);\n+\t\t\ttest_fail();\n+\t\t}\n+\t}\n+\n+\treturn !!ok;\n+}\n+\n+void test__todo_begin(void)\n+{\n+\tassert(ctx.running);\n+\tassert(!ctx.todo);\n+\n+\tctx.todo = 1;\n+}\n+\n+int test__todo_end(const char *location, const char *check, int res)\n+{\n+\tassert(ctx.running);\n+\tassert(ctx.todo);\n+\n+\tctx.todo = 0;\n+\tif (ctx.result == RESULT_SKIP)\n+\t\treturn 1;\n+\tif (res) {\n+\t\ttest_msg(\"todo check '%s' succeeded at %s\", check, location);\n+\t\ttest_fail();\n+\t} else {\n+\t\ttest_todo();\n+\t}\n+\n+\treturn !res;\n+}\n+\n+int check_bool_loc(const char *loc, const char *check, int ok)\n+{\n+\treturn test_assert(loc, check, ok);\n+}\n+\n+union test__tmp test__tmp[2];\n+\n+int check_int_loc(const char *loc, const char *check, int ok,\n+\t\t intmax_t a, intmax_t b)\n+{\n+\tint ret = test_assert(loc, check, ok);\n+\n+\tif (!ret) {\n+\t\ttest_msg(\" left: %\"PRIdMAX, a);\n+\t\ttest_msg(\" right: %\"PRIdMAX, b);\n+\t}\n+\n+\treturn ret;\n+}\n+\n+int check_uint_loc(const char *loc, const char *check, int ok,\n+\t\t uintmax_t a, uintmax_t b)\n+{\n+\tint ret = test_assert(loc, check, ok);\n+\n+\tif (!ret) {\n+\t\ttest_msg(\" left: %\"PRIuMAX, a);\n+\t\ttest_msg(\" right: %\"PRIuMAX, b);\n+\t}\n+\n+\treturn ret;\n+}\n+\n+static void print_one_char(char ch, char quote)\n+{\n+\tif ((unsigned char)ch < 0x20u || ch == 0x7f) {\n+\t\t/* TODO: improve handling of \\a, \\b, \\f ... */\n+\t\tprintf(\"\\\\%03o\", (unsigned char)ch);\n+\t} else {\n+\t\tif (ch == '\\\\' || ch == quote)\n+\t\t\tputc('\\\\', stdout);\n+\t\tputc(ch, stdout);\n+\t}\n+}\n+\n+static void print_char(const char *prefix, char ch)\n+{\n+\tprintf(\"# %s: '\", prefix);\n+\tprint_one_char(ch, '\\'');\n+\tfputs(\"'\\n\", stdout);\n+}\n+\n+int check_char_loc(const char *loc, const char *check, int ok, char a, char b)\n+{\n+\tint ret = test_assert(loc, check, ok);\n+\n+\tif (!ret) {\n+\t\tfflush(stderr);\n+\t\tprint_char(\" left\", a);\n+\t\tprint_char(\" right\", b);\n+\t\tfflush(stdout);\n+\t}\n+\n+\treturn ret;\n+}\n+\n+static void print_str(const char *prefix, const char *str)\n+{\n+\tprintf(\"# %s: \", prefix);\n+\tif (!str) {\n+\t\tfputs(\"NULL\\n\", stdout);\n+\t} else {\n+\t\tputc('\"', stdout);\n+\t\twhile (*str)\n+\t\t\tprint_one_char(*str++, '\"');\n+\t\tfputs(\"\\\"\\n\", stdout);\n+\t}\n+}\n+\n+int check_str_loc(const char *loc, const char *check,\n+\t\t const char *a, const char *b)\n+{\n+\tint ok = (!a && !b) || (a && b && !strcmp(a, b));\n+\tint ret = test_assert(loc, check, ok);\n+\n+\tif (!ret) {\n+\t\tfflush(stderr);\n+\t\tprint_str(\" left\", a);\n+\t\tprint_str(\" right\", b);\n+\t\tfflush(stdout);\n+\t}\n+\n+\treturn ret;\n+}\ndiff --git a/t/unit-tests/test-lib.h b/t/unit-tests/test-lib.h\nnew file mode 100644\nindex 0000000000..8df3804914\n--- /dev/null\n+++ b/t/unit-tests/test-lib.h\n@@ -0,0 +1,143 @@\n+#ifndef TEST_LIB_H\n+#define TEST_LIB_H\n+\n+#include \"git-compat-util.h\"\n+\n+/*\n+ * Run a test function, returns 1 if the test succeeds, 0 if it\n+ * fails. If test_skip_all() has been called then the test will not be\n+ * run. The description for each test should be unique. For example:\n+ *\n+ * TEST(test_something(arg1, arg2), \"something %d %d\", arg1, arg2)\n+ */\n+#define TEST(t, ...)\t\t\t\t\t\\\n+\ttest__run_end(test__run_begin() ? 0 : (t, 1),\t\\\n+\t\t TEST_LOCATION(), __VA_ARGS__)\n+\n+/*\n+ * Print a test plan, should be called before any tests. If the number\n+ * of tests is not known in advance test_done() will automatically\n+ * print a plan at the end of the test program.\n+ */\n+void test_plan(int count);\n+\n+/*\n+ * test_done() must be called at the end of main(). It will print the\n+ * plan if plan() was not called at the beginning of the test program\n+ * and returns the exit code for the test program.\n+ */\n+int test_done(void);\n+\n+/* Skip the current test. */\n+__attribute__((format (printf, 1, 2)))\n+void test_skip(const char *format, ...);\n+\n+/* Skip all remaining tests. */\n+__attribute__((format (printf, 1, 2)))\n+void test_skip_all(const char *format, ...);\n+\n+/* Print a diagnostic message to stdout. */\n+__attribute__((format (printf, 1, 2)))\n+void test_msg(const char *format, ...);\n+\n+/*\n+ * Test checks are built around test_assert(). checks return 1 on\n+ * success, 0 on failure. If any check fails then the test will\n+ * fail. To create a custom check define a function that wraps\n+ * test_assert() and a macro to wrap that function. For example:\n+ *\n+ * static int check_oid_loc(const char *loc, const char *check,\n+ *\t\t\t struct object_id *a, struct object_id *b)\n+ * {\n+ *\t int res = test_assert(loc, check, oideq(a, b));\n+ *\n+ *\t if (res) {\n+ *\t\t test_msg(\" left: %s\", oid_to_hex(a);\n+ *\t\t test_msg(\" right: %s\", oid_to_hex(a);\n+ *\n+ *\t }\n+ *\t return res;\n+ * }\n+ *\n+ * #define check_oid(a, b) \\\n+ *\t check_oid_loc(TEST_LOCATION(), \"oideq(\"#a\", \"#b\")\", a, b)\n+ */\n+int test_assert(const char *location, const char *check, int ok);\n+\n+/* Helper macro to pass the location to checks */\n+#define TEST_LOCATION() TEST__MAKE_LOCATION(__LINE__)\n+\n+/* Check a boolean condition. */\n+#define check(x)\t\t\t\t\\\n+\tcheck_bool_loc(TEST_LOCATION(), #x, x)\n+int check_bool_loc(const char *loc, const char *check, int ok);\n+\n+/*\n+ * Compare two integers. Prints a message with the two values if the\n+ * comparison fails. NB this is not thread safe.\n+ */\n+#define check_int(a, op, b)\t\t\t\t\t\t\\\n+\t(test__tmp[0].i = (a), test__tmp[1].i = (b),\t\t\t\\\n+\t check_int_loc(TEST_LOCATION(), #a\" \"#op\" \"#b,\t\t\t\\\n+\t\t test__tmp[0].i op test__tmp[1].i, a, b))\n+int check_int_loc(const char *loc, const char *check, int ok,\n+\t\t intmax_t a, intmax_t b);\n+\n+/*\n+ * Compare two unsigned integers. Prints a message with the two values\n+ * if the comparison fails. NB this is not thread safe.\n+ */\n+#define check_uint(a, op, b)\t\t\t\t\t\t\\\n+\t(test__tmp[0].u = (a), test__tmp[1].u = (b),\t\t\t\\\n+\t check_uint_loc(TEST_LOCATION(), #a\" \"#op\" \"#b,\t\t\t\\\n+\t\t\ttest__tmp[0].u op test__tmp[1].u, a, b))\n+int check_uint_loc(const char *loc, const char *check, int ok,\n+\t\t uintmax_t a, uintmax_t b);\n+\n+/*\n+ * Compare two chars. Prints a message with the two values if the\n+ * comparison fails. NB this is not thread safe.\n+ */\n+#define check_char(a, op, b)\t\t\t\t\t\t\\\n+\t(test__tmp[0].c = (a), test__tmp[1].c = (b),\t\t\t\\\n+\t check_char_loc(TEST_LOCATION(), #a\" \"#op\" \"#b,\t\t\t\\\n+\t\t\ttest__tmp[0].c op test__tmp[1].c, a, b))\n+int check_char_loc(const char *loc, const char *check, int ok,\n+\t\t char a, char b);\n+\n+/* Check whether two strings are equal. */\n+#define check_str(a, b)\t\t\t\t\t\t\t\\\n+\tcheck_str_loc(TEST_LOCATION(), \"!strcmp(\"#a\", \"#b\")\", a, b)\n+int check_str_loc(const char *loc, const char *check,\n+\t\t const char *a, const char *b);\n+\n+/*\n+ * Wrap a check that is known to fail. If the check succeeds then the\n+ * test will fail. Returns 1 if the check fails, 0 if it\n+ * succeeds. For example:\n+ *\n+ * TEST_TODO(check(0));\n+ */\n+#define TEST_TODO(check) \\\n+\t(test__todo_begin(), test__todo_end(TEST_LOCATION(), #check, check))\n+\n+/* Private helpers */\n+\n+#define TEST__STR(x) #x\n+#define TEST__MAKE_LOCATION(line) __FILE__ \":\" TEST__STR(line)\n+\n+union test__tmp {\n+\tintmax_t i;\n+\tuintmax_t u;\n+\tchar c;\n+};\n+\n+extern union test__tmp test__tmp[2];\n+\n+int test__run_begin(void);\n+__attribute__((format (printf, 3, 4)))\n+int test__run_end(int, const char *, const char *, ...);\n+void test__todo_begin(void);\n+int test__todo_end(const char *, const char *, int);\n+\n+#endif /* TEST_LIB_H */\n-- \n2.42.0.609.gbb76f46606-goog\n\n"},{"id":"482936","messageId":"aa1dfa4892030c2c82170a56f8d7a0a5dd97f103.1696889530.git.steadmon@google.com","threadId":"59937","inReplyTo":"cover.1696889529.git.steadmon@google.com","subject":"[PATCH v8 3/3] ci: run unit tests in CI","fromName":"Josh Steadmon","fromEmail":"steadmon@google.com","sentAt":"2023-10-09T22:21:22Z","receivedAt":"2023-10-09T22:21:45Z","isPatch":true,"sender":{"key":"steadmon@google.com","avatar":"https://avatars.githubusercontent.com/u/2654920?v=4"},"body":"Run unit tests in both Cirrus and GitHub CI. For sharded CI instances\n(currently just Windows on GitHub), run only on the first shard. This is\nOK while we have only a single unit test executable, but we may wish to\ndistribute tests more evenly when we add new unit tests in the future.\n\nWe may also want to add more status output in our unit test framework,\nso that we can do similar post-processing as in\nci/lib.sh:handle_failed_tests().\n\nSigned-off-by: Josh Steadmon \n---\n .cirrus.yml | 2 +-\n ci/run-build-and-tests.sh | 2 ++\n ci/run-test-slice.sh | 5 +++++\n 3 files changed, 8 insertions(+), 1 deletion(-)\n\ndiff --git a/.cirrus.yml b/.cirrus.yml\nindex 4860bebd32..b6280692d2 100644\n--- a/.cirrus.yml\n+++ b/.cirrus.yml\n@@ -19,4 +19,4 @@ freebsd_12_task:\n build_script:\n - su git -c gmake\n test_script:\n- - su git -c 'gmake test'\n+ - su git -c 'gmake DEFAULT_UNIT_TEST_TARGET=unit-tests-prove test unit-tests'\ndiff --git a/ci/run-build-and-tests.sh b/ci/run-build-and-tests.sh\nindex 2528f25e31..7a1466b868 100755\n--- a/ci/run-build-and-tests.sh\n+++ b/ci/run-build-and-tests.sh\n@@ -50,6 +50,8 @@ if test -n \"$run_tests\"\n then\n \tgroup \"Run tests\" make test ||\n \thandle_failed_tests\n+\tgroup \"Run unit tests\" \\\n+\t\tmake DEFAULT_UNIT_TEST_TARGET=unit-tests-prove unit-tests\n fi\n check_unignored_build_artifacts\n \ndiff --git a/ci/run-test-slice.sh b/ci/run-test-slice.sh\nindex a3c67956a8..ae8094382f 100755\n--- a/ci/run-test-slice.sh\n+++ b/ci/run-test-slice.sh\n@@ -15,4 +15,9 @@ group \"Run tests\" make --quiet -C t T=\"$(cd t &&\n \ttr '\\n' ' ')\" ||\n handle_failed_tests\n \n+# We only have one unit test at the moment, so run it in the first slice\n+if [ \"$1\" == \"0\" ] ; then\n+\tgroup \"Run unit tests\" make --quiet -C t unit-tests-prove\n+fi\n+\n check_unignored_build_artifacts\n-- \n2.42.0.609.gbb76f46606-goog\n\n"},{"id":"482942","messageId":"xmqqh6mzwe24.fsf@gitster.g","threadId":"59937","inReplyTo":"cover.1696889529.git.steadmon@google.com","subject":"Re: [PATCH v8 0/3] Add unit test framework and project plan","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2023-10-09T23:50:43Z","receivedAt":"2023-10-09T23:50:56Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Josh Steadmon writes:\n\n> Changes in v8:\n> - Flipped return values for TEST, TEST_TODO, and check_* macros &\n> functions. This makes it easier to reason about control flow for\n> patterns like:\n> if (check(some_condition)) { ... }\n> - Moved unit test binaries to t/unit-tests/bin to simplify .gitignore\n> patterns.\n> - Removed testing of some strbuf implementation details in t-strbuf.c\n>\n>\n> Josh Steadmon (2):\n> unit tests: Add a project plan document\n> ci: run unit tests in CI\n>\n> Phillip Wood (1):\n> unit tests: add TAP unit test framework\n\nThank you, all.\n\nThe other topic to adjust for cmake by Dscho builds on this topic,\nand it needs to be rebased on this updated round. I think I did so\ncorrectly, but because I use neither cmake or Windows, the result is\nnot even compile tested. Sanity checking the result is very much\nappreciated when I push out the result of today's integration cycle.\n\n$ git log --oneline --first-parent --decorate master..js/doc-unit-tests-with-cmake\nd0773c1331 (js/doc-unit-tests-with-cmake) cmake: handle also unit tests\n192de6de57 cmake: use test names instead of full paths\ne1d97bc4df cmake: fix typo in variable name\ne07499b8a7 artifacts-tar: when including `.dll` files, don't forget the unit-tests\n11264f8f42 unit-tests: do show relative file paths\n6c76c5b32d unit-tests: do not mistake `.pdb` files for being executable\n295af2ef26 cmake: also build unit tests\n31c2361349 (js/doc-unit-tests) ci: run unit tests in CI\n3a47942530 unit tests: add TAP unit test framework\neeea7d763a unit tests: add a project plan document\n\n"},{"id":"482953","messageId":"ZSUR+YdzqNTSB0XC@ugly","threadId":"59937","inReplyTo":"81c5148a1267b8f9ce432a950340f0fa16b4d773.1696889530.git.steadmon@google.com","subject":"Re: [PATCH v8 1/3] unit tests: Add a project plan document","fromName":"Oswald Buddenhagen","fromEmail":"oswald.buddenhagen@gmx.de","sentAt":"2023-10-10T08:57:29Z","receivedAt":"2023-10-10T08:57:55Z","isPatch":true,"sender":{"key":"oswald.buddenhagen@gmx.de","avatar":"https://avatars.githubusercontent.com/u/812380?v=4"},"body":"On Mon, Oct 09, 2023 at 03:21:20PM -0700, Josh Steadmon wrote:\n>+=== Comparison\n>+\n>+[format=\"csv\",options=\"header\",width=\"33%\"]\n>+|=====\n>+Framework,\"<>\",\"<>\",\"<>\",\"<>\",\"<>\",\"<>\",\"<>\",\"<>\",\"<>\",\"<>\",\"<>\",\"<>\"\n>\nthe redundancy seems unnecessary; asciidoc should automatically use each \ntarget's section title as the xreflabel.\n\n>+https://lore.kernel.org/git/c902a166-98ce-afba-93f2-ea6027557176@gmail.com/[Custom Git impl.],[lime-background]#GPL v2#,[lime-background]#True#,[lime-background]#True#,[lime-background]#True#,[lime-background]#True#,[lime-background]#True#,[lime-background]#True#,[red-background]#False#,[red-background]#False#,[red-background]#False#,1,0\n>+https://github.com/silentbicycle/greatest[Greatest],[lime-background]#ISC#,[lime-background]#True#,[yellow-background]#Partial#,[lime-background]#True#,[yellow-background]#Partial#,[lime-background]#True#,[lime-background]#True#,[red-background]#False#,[red-background]#False#,[red-background]#False#,3,1400\n>+https://github.com/Snaipe/Criterion[Criterion],[lime-background]#MIT#,[red-background]#False#,[yellow-background]#Partial#,[lime-background]#True#,[lime-background]#True#,[lime-background]#True#,[lime-background]#True#,[lime-background]#True#,[red-background]#False#,[lime-background]#True#,19,1800\n>+https://github.com/rra/c-tap-harness/[C TAP],[lime-background]#Expat#,[lime-background]#True#,[yellow-background]#Partial#,[yellow-background]#Partial#,[lime-background]#True#,[red-background]#False#,[lime-background]#True#,[red-background]#False#,[red-background]#False#,[red-background]#False#,4,33\n>+https://libcheck.github.io/check/[Check],[lime-background]#LGPL v2.1#,[red-background]#False#,[yellow-background]#Partial#,[lime-background]#True#,[lime-background]#True#,[lime-background]#True#,[red-background]#False#,[red-background]#False#,[red-background]#False#,[lime-background]#True#,17,973\n>+|=====\n>+\ni find this totally unreadable in its raw form.\nconsider user-defined document-attributes for specific cell contents.\nexternalizing the urls would probably help as well (i'm not sure how to \ndo that best).\n\nregards\n"},{"id":"483088","messageId":"ZScQG5QHznMEGzhC@google.com","threadId":"59937","inReplyTo":"ZSUR+YdzqNTSB0XC@ugly","subject":"Re: [PATCH v8 1/3] unit tests: Add a project plan document","fromName":"Josh Steadmon","fromEmail":"steadmon@google.com","sentAt":"2023-10-11T21:14:03Z","receivedAt":"2023-10-11T21:14:11Z","isPatch":true,"sender":{"key":"steadmon@google.com","avatar":"https://avatars.githubusercontent.com/u/2654920?v=4"},"body":"On 2023.10.10 10:57, Oswald Buddenhagen wrote:\n> On Mon, Oct 09, 2023 at 03:21:20PM -0700, Josh Steadmon wrote:\n> > +=== Comparison\n> > +\n> > +[format=\"csv\",options=\"header\",width=\"33%\"]\n> > +|=====\n> > +Framework,\"<>\",\"<>\",\"<>\",\"<>\",\"<>\",\"<>\",\"<>\",\"<>\",\"<>\",\"<>\",\"<>\",\"<>\"\n> > \n> the redundancy seems unnecessary; asciidoc should automatically use each\n> target's section title as the xreflabel.\n\nHmm, this doesn't seem to work for me. It only renders as\n\"[anchor-label]\".\n\n\n> > +https://lore.kernel.org/git/c902a166-98ce-afba-93f2-ea6027557176@gmail.com/[Custom Git impl.],[lime-background]#GPL v2#,[lime-background]#True#,[lime-background]#True#,[lime-background]#True#,[lime-background]#True#,[lime-background]#True#,[lime-background]#True#,[red-background]#False#,[red-background]#False#,[red-background]#False#,1,0\n> > +https://github.com/silentbicycle/greatest[Greatest],[lime-background]#ISC#,[lime-background]#True#,[yellow-background]#Partial#,[lime-background]#True#,[yellow-background]#Partial#,[lime-background]#True#,[lime-background]#True#,[red-background]#False#,[red-background]#False#,[red-background]#False#,3,1400\n> > +https://github.com/Snaipe/Criterion[Criterion],[lime-background]#MIT#,[red-background]#False#,[yellow-background]#Partial#,[lime-background]#True#,[lime-background]#True#,[lime-background]#True#,[lime-background]#True#,[lime-background]#True#,[red-background]#False#,[lime-background]#True#,19,1800\n> > +https://github.com/rra/c-tap-harness/[C TAP],[lime-background]#Expat#,[lime-background]#True#,[yellow-background]#Partial#,[yellow-background]#Partial#,[lime-background]#True#,[red-background]#False#,[lime-background]#True#,[red-background]#False#,[red-background]#False#,[red-background]#False#,4,33\n> > +https://libcheck.github.io/check/[Check],[lime-background]#LGPL v2.1#,[red-background]#False#,[yellow-background]#Partial#,[lime-background]#True#,[lime-background]#True#,[lime-background]#True#,[red-background]#False#,[red-background]#False#,[red-background]#False#,[lime-background]#True#,17,973\n> > +|=====\n> > +\n> i find this totally unreadable in its raw form.\n> consider user-defined document-attributes for specific cell contents.\n> externalizing the urls would probably help as well (i'm not sure how to do\n> that best).\n\nAh yeah, user-defined attributes definitely cleans this up quite a bit.\nThanks for the tip!\n"},{"id":"483093","messageId":"xmqqwmvskf8t.fsf@gitster.g","threadId":"59937","inReplyTo":"00d3c95a81449bf49c4ce992d862d7a858691840.1696889530.git.steadmon@google.com","subject":"Re: [PATCH v8 2/3] unit tests: add TAP unit test framework","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2023-10-11T21:42:42Z","receivedAt":"2023-10-11T21:42:52Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Josh Steadmon writes:\n\n> +\t/* Initialized strbufs should always have a non-NULL buffer */\n> +\tif (buf->buf == NULL)\n> +\t\treturn 0;\n\nThis upsets Coccinelle (equals-null). I'll queue this on top for\nnow to work around CI breakage.\n\nThanks.\n\n----- >8 -----\nSubject: [PATCH] SQUASH???\n\nCoccinelle suggested style fix.\n---\n t/unit-tests/t-strbuf.c | 2 +-\n 1 file changed, 1 insertion(+), 1 deletion(-)\n\ndiff --git a/t/unit-tests/t-strbuf.c b/t/unit-tests/t-strbuf.c\nindex c2fcb0cbd6..8388442426 100644\n--- a/t/unit-tests/t-strbuf.c\n+++ b/t/unit-tests/t-strbuf.c\n@@ -28,7 +28,7 @@ static void setup_populated(void (*f)(struct strbuf*, void*), char *init_str, vo\n static int assert_sane_strbuf(struct strbuf *buf)\n {\n \t/* Initialized strbufs should always have a non-NULL buffer */\n-\tif (buf->buf == NULL)\n+\tif (!buf->buf)\n \t\treturn 0;\n \t/* Buffers should always be NUL-terminated */\n \tif (buf->buf[buf->len] != '\\0')\n-- \n2.42.0-345-gaab89be2eb\n\n"},{"id":"483111","messageId":"ZScqLzGiDPZvLh2k@ugly","threadId":"59937","inReplyTo":"ZScQG5QHznMEGzhC@google.com","subject":"Re: [PATCH v8 1/3] unit tests: Add a project plan document","fromName":"Oswald Buddenhagen","fromEmail":"oswald.buddenhagen@gmx.de","sentAt":"2023-10-11T23:05:19Z","receivedAt":"2023-10-11T23:05:28Z","isPatch":true,"sender":{"key":"oswald.buddenhagen@gmx.de","avatar":"https://avatars.githubusercontent.com/u/812380?v=4"},"body":"On Wed, Oct 11, 2023 at 02:14:03PM -0700, Josh Steadmon wrote:\n>On 2023.10.10 10:57, Oswald Buddenhagen wrote:\n>> On Mon, Oct 09, 2023 at 03:21:20PM -0700, Josh Steadmon wrote:\n>> > +=== Comparison\n>> > +\n>> > +[format=\"csv\",options=\"header\",width=\"33%\"]\n>> > +|=====\n>> > +Framework,\"<>\",\"<>\",\"<>\",\"<>\",\"<>\",\"<>\",\"<>\",\"<>\",\"<>\",\"<>\",\"<>\",\"<>\"\n>> > \n>> the redundancy seems unnecessary; asciidoc should automatically use each\n>> target's section title as the xreflabel.\n>\n>Hmm, this doesn't seem to work for me. It only renders as\n>\"[anchor-label]\".\n>\ni thought\nhttps://docs.asciidoctor.org/asciidoc/latest/attributes/id/#customize-automatic-xreftext \nis pretty clear about it, though. maybe the actual tooling uses an older \nversion of the spec? or is buggy? or the placement of the titles is \nincorrect? or this applies to different links or targets only? or am i \nmisreading something? or ...?\n\nregards\n"},{"id":"483299","messageId":"93a18989-bf05-4318-8d85-cf23c0f32170@gmail.com","threadId":"59937","inReplyTo":"cover.1696889529.git.steadmon@google.com","subject":"Re: [PATCH v8 0/3] Add unit test framework and project plan","fromName":"","fromEmail":"phillip.wood123@gmail.com","sentAt":"2023-10-16T10:07:02Z","receivedAt":"2023-10-16T10:07:07Z","isPatch":true,"sender":{"key":"phillip.wood@dunelm.org.uk","avatar":null},"body":"Hi Josh\n\nThanks for the update\n\nOn 09/10/2023 23:21, Josh Steadmon wrote:\n> In addition to reviewing the patches in this series, reviewers can help\n> this series progress by chiming in on these remaining TODOs:\n> - Figure out if we should split t-basic.c into multiple meta-tests, to\n> avoid merge conflicts and changes to expected text in\n> t0080-unit-test-output.sh.\n\nI think it depends on how many new tests we think we're going to want to \nadd here. I can see us adding a few more check_* macros (comparing \nobject ids and arrays of bytes spring to mind) and wanting to test them \nhere, but (perhaps naïvely) I don't expect huge amounts of churn here.\n\n> - Figure out if we should de-duplicate assertions in t-strbuf.c at the\n> cost of making tests less self-contained and diagnostic output less\n> helpful.\n\nIn principle we could pass the location information along to any helper \nfunction, I'm not sure how easy that is at the moment. We can get \nreasonable error messages by using the check*() macros in the helper and \nwrapping the call to the helper with check() as well. For example\n\nstatic int assert_sane_strbuf(struct strbuf *buf)\n{\n\t/* Initialized strbufs should always have a non-NULL buffer */\n\tif (!check(!!buf->buf))\n\t\treturn 0;\n\t/* Buffers should always be NUL-terminated */\n\tif (!check_char(buf->buf[buf->len], ==, '\\0'))\n\t\treturn 0;\n\t/*\n\t * Freshly-initialized strbufs may not have a dynamically allocated\n\t * buffer\n\t */\n\tif (buf->len == 0 && buf->alloc == 0)\n\t\treturn 1;\n\t/* alloc must be at least one byte larger than len */\n\treturn check_uint(buf->len, <, buf->alloc);\n}\n\nand in the test function call it as\n\n\tcheck(assert_sane_strbuf(buf));\n\nwhich gives error messages like\n\n# check \"buf->len < buf->alloc\" failed at t/unit-tests/t-strbuf.c:43\n# left: 5\n# right: 0\n# check \"assert_sane_strbuf(&buf)\" failed at t/unit-tests/t-strbuf.c:60\n\nSo we can see where assert_sane_strbuf() was called and which assertion \nin assert_sane_strbuf() failed.\n\n> - Figure out if we should collect unit tests statistics similar to the\n> \"counts\" files for shell tests\n\nUnless someone has an immediate need for that I'd be tempted to leave it \nwait until someone requests that data.\n\n> - Decide if it's OK to wait on sharding unit tests across \"sliced\" CI\n> instances\n\nHopefully the unit tests will run fast enough that we don't need to \nworry about that in the early stages.\n\n> - Provide guidelines for writing new unit tests\n\nThis is not a comprehensive list but we should recommend that\n\n- tests avoid leaking resources so the leak sanitizer see if the code\n being tested has a resource leak.\n\n- tests check that pointers are not NULL before deferencing them to\n avoid the whole program being taken down with SIGSEGV.\n\n- tests are written with easy debugging in mind - i.e. good diagnostic\n messages. Hopefully the check* macros make that easy to do.\n\n> Changes in v8:\n> - Flipped return values for TEST, TEST_TODO, and check_* macros &\n> functions. This makes it easier to reason about control flow for\n> patterns like:\n> if (check(some_condition)) { ... } > - Moved unit test binaries to t/unit-tests/bin to simplify .gitignore\n> patterns.\n\nThanks for the updates to the test library, the range diff looks good to me.\n\n > - Removed testing of some strbuf implementation details in t-strbuf.c\n\nI agree that makes sense. I think it would be good to update \nassert_sane_strbuf() to use the check* macros as suggest above.\n\nBest Wishes\n\nPhillip\n"},{"id":"483306","messageId":"20231016134421.21659-1-phillip.wood123@gmail.com","threadId":"59937","inReplyTo":"00d3c95a81449bf49c4ce992d862d7a858691840.1696889530.git.steadmon@google.com","subject":"[PATCH v8 2.5/3] fixup! unit tests: add TAP unit test framework","fromName":"Phillip Wood","fromEmail":"phillip.wood123@gmail.com","sentAt":"2023-10-16T13:43:00Z","receivedAt":"2023-10-16T13:44:43Z","isPatch":true,"sender":{"key":"phillip.wood@dunelm.org.uk","avatar":null},"body":"From: Phillip Wood \n\nHere are a couple of cleanups for the unit test framework that I\nnoticed.\n\nUpdate the documentation of the example custom check to reflect the\nchange in return value of test_assert() and mention that\nchecks should be careful when dereferencing pointer arguments.\n\nAlso avoid evaluating macro augments twice in check_int() and\nfriends. The global variable test__tmp was introduced to avoid\nevaluating the arguments to these macros more than once but the macros\nfailed to use it when passing the values being compared to\ncheck_int_loc().\n\nSigned-off-by: Phillip Wood \n---\n t/unit-tests/test-lib.h | 26 ++++++++++++++++----------\n 1 file changed, 16 insertions(+), 10 deletions(-)\n\ndiff --git a/t/unit-tests/test-lib.h b/t/unit-tests/test-lib.h\nindex 8df3804914..a8f07ae0b7 100644\n--- a/t/unit-tests/test-lib.h\n+++ b/t/unit-tests/test-lib.h\n@@ -42,18 +42,21 @@ void test_msg(const char *format, ...);\n \n /*\n * Test checks are built around test_assert(). checks return 1 on\n- * success, 0 on failure. If any check fails then the test will\n- * fail. To create a custom check define a function that wraps\n- * test_assert() and a macro to wrap that function. For example:\n+ * success, 0 on failure. If any check fails then the test will fail. To\n+ * create a custom check define a function that wraps test_assert() and\n+ * a macro to wrap that function to provide a source location and\n+ * stringified arguments. Custom checks that take pointer arguments\n+ * should be careful to check that they are non-NULL before\n+ * dereferencing them. For example:\n *\n * static int check_oid_loc(const char *loc, const char *check,\n *\t\t\t struct object_id *a, struct object_id *b)\n * {\n- *\t int res = test_assert(loc, check, oideq(a, b));\n+ *\t int res = test_assert(loc, check, a && b && oideq(a, b));\n *\n- *\t if (res) {\n- *\t\t test_msg(\" left: %s\", oid_to_hex(a);\n- *\t\t test_msg(\" right: %s\", oid_to_hex(a);\n+ *\t if (!res) {\n+ *\t\t test_msg(\" left: %s\", a ? oid_to_hex(a) : \"NULL\";\n+ *\t\t test_msg(\" right: %s\", b ? oid_to_hex(a) : \"NULL\";\n *\n *\t }\n *\t return res;\n@@ -79,7 +82,8 @@ int check_bool_loc(const char *loc, const char *check, int ok);\n #define check_int(a, op, b)\t\t\t\t\t\t\\\n \t(test__tmp[0].i = (a), test__tmp[1].i = (b),\t\t\t\\\n \t check_int_loc(TEST_LOCATION(), #a\" \"#op\" \"#b,\t\t\t\\\n-\t\t test__tmp[0].i op test__tmp[1].i, a, b))\n+\t\t test__tmp[0].i op test__tmp[1].i,\t\t\\\n+\t\t test__tmp[0].i, test__tmp[1].i))\n int check_int_loc(const char *loc, const char *check, int ok,\n \t\t intmax_t a, intmax_t b);\n \n@@ -90,7 +94,8 @@ int check_int_loc(const char *loc, const char *check, int ok,\n #define check_uint(a, op, b)\t\t\t\t\t\t\\\n \t(test__tmp[0].u = (a), test__tmp[1].u = (b),\t\t\t\\\n \t check_uint_loc(TEST_LOCATION(), #a\" \"#op\" \"#b,\t\t\t\\\n-\t\t\ttest__tmp[0].u op test__tmp[1].u, a, b))\n+\t\t\ttest__tmp[0].u op test__tmp[1].u,\t\t\\\n+\t\t\ttest__tmp[0].u, test__tmp[1].u))\n int check_uint_loc(const char *loc, const char *check, int ok,\n \t\t uintmax_t a, uintmax_t b);\n \n@@ -101,7 +106,8 @@ int check_uint_loc(const char *loc, const char *check, int ok,\n #define check_char(a, op, b)\t\t\t\t\t\t\\\n \t(test__tmp[0].c = (a), test__tmp[1].c = (b),\t\t\t\\\n \t check_char_loc(TEST_LOCATION(), #a\" \"#op\" \"#b,\t\t\t\\\n-\t\t\ttest__tmp[0].c op test__tmp[1].c, a, b))\n+\t\t\ttest__tmp[0].c op test__tmp[1].c,\t\t\\\n+\t\t\ttest__tmp[0].c, test__tmp[1].c))\n int check_char_loc(const char *loc, const char *check, int ok,\n \t\t char a, char b);\n \n-- \n2.42.0.506.g0dd4464cfd3\n\n"},{"id":"483312","messageId":"xmqq1qduo6yr.fsf@gitster.g","threadId":"59937","inReplyTo":"20231016134421.21659-1-phillip.wood123@gmail.com","subject":"Re: [PATCH v8 2.5/3] fixup! unit tests: add TAP unit test framework","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2023-10-16T16:41:32Z","receivedAt":"2023-10-16T16:42:03Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Phillip Wood writes:\n\n> From: Phillip Wood \n>\n> Here are a couple of cleanups for the unit test framework that I\n> noticed.\n\nThanks. I trust that this will be squashed into the next update,\nbut in the meantime, I'll include it in the copy of the series I\nhave (without squashing). Here is another one I noticed.\n\n----- >8 --------- >8 --------- >8 -----\nSubject: [PATCH] fixup! ci: run unit tests in CI\n\nA CI job failed due to contrib/coccinelle/equals-null.cocci\nand suggested this change, which seems sensible.\n\nSigned-off-by: Junio C Hamano \n---\n t/unit-tests/t-strbuf.c | 2 +-\n 1 file changed, 1 insertion(+), 1 deletion(-)\n\ndiff --git a/t/unit-tests/t-strbuf.c b/t/unit-tests/t-strbuf.c\nindex c2fcb0cbd6..8388442426 100644\n--- a/t/unit-tests/t-strbuf.c\n+++ b/t/unit-tests/t-strbuf.c\n@@ -28,7 +28,7 @@ static void setup_populated(void (*f)(struct strbuf*, void*), char *init_str, vo\n static int assert_sane_strbuf(struct strbuf *buf)\n {\n \t/* Initialized strbufs should always have a non-NULL buffer */\n-\tif (buf->buf == NULL)\n+\tif (!buf->buf)\n \t\treturn 0;\n \t/* Buffers should always be NUL-terminated */\n \tif (buf->buf[buf->len] != '\\0')\n-- \n2.42.0-398-ga9ecda2788\n\n"},{"id":"483491","messageId":"20231019152726.14624-1-phillip.wood123@gmail.com","threadId":"59937","inReplyTo":"xmqqh6mzwe24.fsf@gitster.g","subject":"[PATCH 0/3] CMake unit test fixups","fromName":"Phillip Wood","fromEmail":"phillip.wood123@gmail.com","sentAt":"2023-10-19T15:21:48Z","receivedAt":"2023-10-19T15:27:49Z","isPatch":true,"sender":{"key":"phillip.wood@dunelm.org.uk","avatar":null},"body":"From: Phillip Wood \n\nHi Junio\n\n> The other topic to adjust for cmake by Dscho builds on this topic,\n> and it needs to be rebased on this updated round. I think I did so\n> correctly, but because I use neither cmake or Windows, the result is\n> not even compile tested. Sanity checking the result is very much\n> appreciated when I push out the result of today's integration cycle.\n\nI need these fixups to get our CI to successfully build an run the\nunit tests using CMake & MSVC. They are all adjusting paths now that\nthe unit test programs are built in t/unit-tests/bin\n\nYou can see the unit tests passing at\nhttps://github.com/phillipwood/git/actions/runs/6575606322/job/17863460719\nNote that I have split up the patches since that run but the changes\nare the same.\n\nBest Wishes\n\nPhillip\n\n\nPhillip Wood (3):\n fixup! cmake: also build unit tests\n fixup! artifacts-tar: when including `.dll` files, don't forget the\n unit-tests\n fixup! cmake: handle also unit tests\n\n Makefile | 2 +-\n contrib/buildsystems/CMakeLists.txt | 8 ++++----\n 2 files changed, 5 insertions(+), 5 deletions(-)\n\n-- \n2.42.0.506.g0dd4464cfd3\n\n"},{"id":"483490","messageId":"20231019152726.14624-2-phillip.wood123@gmail.com","threadId":"59937","inReplyTo":"20231019152726.14624-1-phillip.wood123@gmail.com","subject":"[PATCH 1/3] fixup! cmake: also build unit tests","fromName":"Phillip Wood","fromEmail":"phillip.wood123@gmail.com","sentAt":"2023-10-19T15:21:49Z","receivedAt":"2023-10-19T15:27:50Z","isPatch":true,"sender":{"key":"phillip.wood@dunelm.org.uk","avatar":null},"body":"From: Phillip Wood \n\n---\n contrib/buildsystems/CMakeLists.txt | 6 +++---\n 1 file changed, 3 insertions(+), 3 deletions(-)\n\ndiff --git a/contrib/buildsystems/CMakeLists.txt b/contrib/buildsystems/CMakeLists.txt\nindex d21835ca65..20f38e94c9 100644\n--- a/contrib/buildsystems/CMakeLists.txt\n+++ b/contrib/buildsystems/CMakeLists.txt\n@@ -973,12 +973,12 @@ foreach(unit_test ${unit_test_PROGRAMS})\n \tadd_executable(\"${unit_test}\" \"${CMAKE_SOURCE_DIR}/t/unit-tests/${unit_test}.c\")\n \ttarget_link_libraries(\"${unit_test}\" unit-test-lib common-main)\n \tset_target_properties(\"${unit_test}\"\n-\t\tPROPERTIES RUNTIME_OUTPUT_DIRECTORY ${CMAKE_BINARY_DIR}/t/unit-tests)\n+\t\tPROPERTIES RUNTIME_OUTPUT_DIRECTORY ${CMAKE_BINARY_DIR}/t/unit-tests/bin)\n \tif(MSVC)\n \t\tset_target_properties(\"${unit_test}\"\n-\t\t\tPROPERTIES RUNTIME_OUTPUT_DIRECTORY_DEBUG ${CMAKE_BINARY_DIR}/t/unit-tests)\n+\t\t\tPROPERTIES RUNTIME_OUTPUT_DIRECTORY_DEBUG ${CMAKE_BINARY_DIR}/t/unit-tests/bin)\n \t\tset_target_properties(\"${unit_test}\"\n-\t\t\tPROPERTIES RUNTIME_OUTPUT_DIRECTORY_RELEASE ${CMAKE_BINARY_DIR}/t/unit-tests)\n+\t\t\tPROPERTIES RUNTIME_OUTPUT_DIRECTORY_RELEASE ${CMAKE_BINARY_DIR}/t/unit-tests/bin)\n \tendif()\n \tlist(APPEND PROGRAMS_BUILT \"${unit_test}\")\n \n-- \n2.42.0.506.g0dd4464cfd3\n\n"},{"id":"483492","messageId":"20231019152726.14624-3-phillip.wood123@gmail.com","threadId":"59937","inReplyTo":"20231019152726.14624-1-phillip.wood123@gmail.com","subject":"[PATCH 2/3] fixup! artifacts-tar: when including `.dll` files, don't forget the unit-tests","fromName":"Phillip Wood","fromEmail":"phillip.wood123@gmail.com","sentAt":"2023-10-19T15:21:50Z","receivedAt":"2023-10-19T15:27:50Z","isPatch":true,"sender":{"key":"phillip.wood@dunelm.org.uk","avatar":null},"body":"From: Phillip Wood \n\n---\n Makefile | 2 +-\n 1 file changed, 1 insertion(+), 1 deletion(-)\n\ndiff --git a/Makefile b/Makefile\nindex 075d8e4899..e15c34e506 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -3597,7 +3597,7 @@ rpm::\n .PHONY: rpm\n \n ifneq ($(INCLUDE_DLLS_IN_ARTIFACTS),)\n-OTHER_PROGRAMS += $(shell echo *.dll t/helper/*.dll t/unit-tests/*.dll)\n+OTHER_PROGRAMS += $(shell echo *.dll t/helper/*.dll t/unit-tests/bin/*.dll)\n endif\n \n artifacts-tar:: $(ALL_COMMANDS_TO_INSTALL) $(SCRIPT_LIB) $(OTHER_PROGRAMS) \\\n-- \n2.42.0.506.g0dd4464cfd3\n\n"},{"id":"483493","messageId":"20231019152726.14624-4-phillip.wood123@gmail.com","threadId":"59937","inReplyTo":"20231019152726.14624-1-phillip.wood123@gmail.com","subject":"[PATCH 3/3] fixup! cmake: handle also unit tests","fromName":"Phillip Wood","fromEmail":"phillip.wood123@gmail.com","sentAt":"2023-10-19T15:21:51Z","receivedAt":"2023-10-19T15:27:51Z","isPatch":true,"sender":{"key":"phillip.wood@dunelm.org.uk","avatar":null},"body":"From: Phillip Wood \n\n---\n contrib/buildsystems/CMakeLists.txt | 2 +-\n 1 file changed, 1 insertion(+), 1 deletion(-)\n\ndiff --git a/contrib/buildsystems/CMakeLists.txt b/contrib/buildsystems/CMakeLists.txt\nindex 20f38e94c9..671c7ead75 100644\n--- a/contrib/buildsystems/CMakeLists.txt\n+++ b/contrib/buildsystems/CMakeLists.txt\n@@ -990,7 +990,7 @@ foreach(unit_test ${unit_test_PROGRAMS})\n \tif(NOT ${unit_test} STREQUAL \"t-basic\")\n \t\tadd_test(NAME \"t.unit-tests.${unit_test}\"\n \t\t\tCOMMAND \"./${unit_test}\"\n-\t\t\tWORKING_DIRECTORY ${CMAKE_SOURCE_DIR}/t/unit-tests)\n+\t\t\tWORKING_DIRECTORY ${CMAKE_SOURCE_DIR}/t/unit-tests/bin)\n \tendif()\n endforeach()\n \n-- \n2.42.0.506.g0dd4464cfd3\n\n"},{"id":"483516","messageId":"xmqqfs26whb5.fsf@gitster.g","threadId":"59937","inReplyTo":"20231019152726.14624-1-phillip.wood123@gmail.com","subject":"Re: [PATCH 0/3] CMake unit test fixups","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2023-10-19T19:19:58Z","receivedAt":"2023-10-19T19:20:05Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Phillip Wood writes:\n\n> I need these fixups to get our CI to successfully build an run the\n> unit tests using CMake & MSVC. They are all adjusting paths now that\n> the unit test programs are built in t/unit-tests/bin\n\nThanks! Very much appreciated.\n"},{"id":"484003","messageId":"CAP8UFD26X4MPbJs4KfNOgicLMb-wiuFZj3Hw17acMmmc_=vcqQ@mail.gmail.com","threadId":"59937","inReplyTo":"81c5148a1267b8f9ce432a950340f0fa16b4d773.1696889530.git.steadmon@google.com","subject":"Re: [PATCH v8 1/3] unit tests: Add a project plan document","fromName":"Christian Couder","fromEmail":"christian.couder@gmail.com","sentAt":"2023-10-27T20:12:12Z","receivedAt":"2023-10-27T20:12:29Z","isPatch":true,"sender":{"key":"christian.couder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/208954?v=4"},"body":"On Tue, Oct 10, 2023 at 12:22 AM Josh Steadmon wrote:\n>\n> In our current testing environment, we spend a significant amount of\n> effort crafting end-to-end tests for error conditions that could easily\n> be captured by unit tests (or we simply forgo some hard-to-setup and\n> rare error conditions). Describe what we hope to accomplish by\n> implementing unit tests, and explain some open questions and milestones.\n> Discuss desired features for test frameworks/harnesses, and provide a\n> preliminary comparison of several different frameworks.\n\nNit: Not sure why the test framework comparison is \"preliminary\" as we\nhave actually selected a unit test framework and are adding it in the\nnext patch of the series. I understand that this was perhaps written\nbefore the choice was made, but maybe we might want to update that\nnow.\n\n> diff --git a/Documentation/technical/unit-tests.txt b/Documentation/technical/unit-tests.txt\n> new file mode 100644\n> index 0000000000..b7a89cc838\n> --- /dev/null\n> +++ b/Documentation/technical/unit-tests.txt\n> @@ -0,0 +1,220 @@\n> += Unit Testing\n> +\n> +In our current testing environment, we spend a significant amount of effort\n> +crafting end-to-end tests for error conditions that could easily be captured by\n> +unit tests (or we simply forgo some hard-to-setup and rare error conditions).\n> +Unit tests additionally provide stability to the codebase and can simplify\n> +debugging through isolation. Writing unit tests in pure C, rather than with our\n> +current shell/test-tool helper setup, simplifies test setup, simplifies passing\n> +data around (no shell-isms required), and reduces testing runtime by not\n> +spawning a separate process for every test invocation.\n> +\n> +We believe that a large body of unit tests, living alongside the existing test\n> +suite, will improve code quality for the Git project.\n\nI agree with that.\n\n> +== Choosing a framework\n> +\n> +We believe the best option is to implement a custom TAP framework for the Git\n> +project. We use a version of the framework originally proposed in\n> +https://lore.kernel.org/git/c902a166-98ce-afba-93f2-ea6027557176@gmail.com/[1].\n\nNit: Logically I would think that our opinion should come after the\ncomparison and be backed by it.\n\n> +== Choosing a test harness\n> +\n> +During upstream discussion, it was occasionally noted that `prove` provides many\n> +convenient features, such as scheduling slower tests first, or re-running\n> +previously failed tests.\n> +\n> +While we already support the use of `prove` as a test harness for the shell\n> +tests, it is not strictly required. The t/Makefile allows running shell tests\n> +directly (though with interleaved output if parallelism is enabled). Git\n> +developers who wish to use `prove` as a more advanced harness can do so by\n> +setting DEFAULT_TEST_TARGET=prove in their config.mak.\n> +\n> +We will follow a similar approach for unit tests: by default the test\n> +executables will be run directly from the t/Makefile, but `prove` can be\n> +configured with DEFAULT_UNIT_TEST_TARGET=prove.\n\nNice that it can be used.\n\nThe rest of the file looks good.\n"},{"id":"484004","messageId":"CAP8UFD3eY_i36YO0OcpAp9ey5KO0q-PrwvjSLRXKYQb=iZ8JCA@mail.gmail.com","threadId":"59937","inReplyTo":"00d3c95a81449bf49c4ce992d862d7a858691840.1696889530.git.steadmon@google.com","subject":"Re: [PATCH v8 2/3] unit tests: add TAP unit test framework","fromName":"Christian Couder","fromEmail":"christian.couder@gmail.com","sentAt":"2023-10-27T20:15:08Z","receivedAt":"2023-10-27T20:15:23Z","isPatch":true,"sender":{"key":"christian.couder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/208954?v=4"},"body":"On Tue, Oct 10, 2023 at 12:22 AM Josh Steadmon wrote:\n>\n> From: Phillip Wood \n>\n> This patch contains an implementation for writing unit tests with TAP\n> output. Each test is a function that contains one or more checks. The\n> test is run with the TEST() macro and if any of the checks fail then the\n> test will fail. A complete program that tests STRBUF_INIT would look\n> like\n>\n> #include \"test-lib.h\"\n> #include \"strbuf.h\"\n>\n> static void t_static_init(void)\n> {\n> struct strbuf buf = STRBUF_INIT;\n>\n> check_uint(buf.len, ==, 0);\n> check_uint(buf.alloc, ==, 0);\n> check_char(buf.buf[0], ==, '\\0');\n> }\n>\n> int main(void)\n> {\n> TEST(t_static_init(), \"static initialization works);\n>\n> return test_done();\n> }\n>\n> The output of this program would be\n>\n> ok 1 - static initialization works\n> 1..1\n>\n> If any of the checks in a test fail then they print a diagnostic message\n> to aid debugging and the test will be reported as failing. For example a\n> failing integer check would look like\n>\n> # check \"x >= 3\" failed at my-test.c:102\n\nI wonder if it would be a bit better to say that the test was an\ninteger test for example with \"check_int(x >= 3) failed ...\"\n\n> # left: 2\n> # right: 3\n\nI like \"expected\" and \"actual\" better than \"left\" and \"right\", not\nsure how it's possible to have that in a way consistent with the shell\ntests though.\n\n> not ok 1 - x is greater than or equal to three\n>\n> There are a number of check functions implemented so far. check() checks\n> a boolean condition, check_int(), check_uint() and check_char() take two\n> values to compare and a comparison operator. check_str() will check if\n> two strings are equal. Custom checks are simple to implement as shown in\n> the comments above test_assert() in test-lib.h.\n\nYeah, nice.\n\n> Tests can be skipped with test_skip() which can be supplied with a\n> reason for skipping which it will print. Tests can print diagnostic\n> messages with test_msg(). Checks that are known to fail can be wrapped\n> in TEST_TODO().\n\nMaybe TEST_TOFIX() would be a bit more clear, but \"TODO\" is something\nthat is more likely to be searched for than \"TOFIX\", so Ok.\n\n> There are a couple of example test programs included in this\n> patch. t-basic.c implements some self-tests and demonstrates the\n> diagnostic output for failing test. The output of this program is\n> checked by t0080-unit-test-output.sh. t-strbuf.c shows some example\n> unit tests for strbuf.c\n>\n> The unit tests will be built as part of the default \"make all\" target,\n> to avoid bitrot. If you wish to build just the unit tests, you can run\n> \"make build-unit-tests\". To run the tests, you can use \"make unit-tests\"\n> or run the test binaries directly, as in \"./t/unit-tests/bin/t-strbuf\".\n\nNice!\n\n> +unit-tests-prove:\n> + @echo \"*** prove - unit tests ***\"; $(PROVE) $(GIT_PROVE_OPTS) $(UNIT_TESTS)\n\nNice, but DEFAULT_TEST_TARGET=prove isn't used. So not sure how\nimportant or relevant the 'prove' related sections are in the\nDocumentation/technical/unit-tests.txt file introduced by the previous\npatch.\n\n\n> +int test_assert(const char *location, const char *check, int ok)\n> +{\n> + assert(ctx.running);\n> +\n> + if (ctx.result == RESULT_SKIP) {\n> + test_msg(\"skipping check '%s' at %s\", check, location);\n> + return 1;\n> + } else if (!ctx.todo) {\n\nI think it would be a bit clearer without the \"else\" above and with\nthe \"if (!ctx.todo) {\" starting on a new line.\n\n> + if (ok) {\n> + test_pass();\n> + } else {\n> + test_msg(\"check \\\"%s\\\" failed at %s\", check, location);\n> + test_fail();\n> + }\n> + }\n> +\n> + return !!ok;\n> +}\n\nOtherwise it looks good to me.\n"},{"id":"484005","messageId":"CAP8UFD0hBy=FufxiKfbUr=0M5H-oL4SDk=zKYRLw-vwq6+G05A@mail.gmail.com","threadId":"59937","inReplyTo":"cover.1696889529.git.steadmon@google.com","subject":"Re: [PATCH v8 0/3] Add unit test framework and project plan","fromName":"Christian Couder","fromEmail":"christian.couder@gmail.com","sentAt":"2023-10-27T20:26:01Z","receivedAt":"2023-10-27T20:26:17Z","isPatch":true,"sender":{"key":"christian.couder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/208954?v=4"},"body":"On Tue, Oct 10, 2023 at 12:22 AM Josh Steadmon wrote:\n>\n> In our current testing environment, we spend a significant amount of\n> effort crafting end-to-end tests for error conditions that could easily\n> be captured by unit tests (or we simply forgo some hard-to-setup and\n> rare error conditions). Unit tests additionally provide stability to the\n> codebase and can simplify debugging through isolation. Turning parts of\n> Git into libraries[1] gives us the ability to run unit tests on the\n> libraries and to write unit tests in C. Writing unit tests in pure C,\n> rather than with our current shell/test-tool helper setup, simplifies\n> test setup, simplifies passing data around (no shell-isms required), and\n> reduces testing runtime by not spawning a separate process for every\n> test invocation.\n>\n> This series begins with a project document covering our goals for adding\n> unit tests and a discussion of alternative frameworks considered, as\n> well as the features used to evaluate them. A rendered preview of this\n> doc can be found at [2]. It also adds Phillip Wood's TAP implemenation\n> (with some slightly re-worked Makefile rules) and a sample strbuf unit\n> test. Finally, we modify the configs for GitHub and Cirrus CI to run the\n> unit tests. Sample runs showing successful CI runs can be found at [3],\n> [4], and [5].\n\nI took a look at this version and I like it very much. I left a few\ncomments. My only real wish would be to use something like \"actual\"\nand \"expected\" instead of \"left\" and \"right\" in case of test failure\nand perhaps in the argument names of functions and macros. Not sure it\nis easy to do in the same way as in the shell framework though.\n\nThanks!\n"},{"id":"484304","messageId":"ZUKLhInUh19g11Dl@google.com","threadId":"59937","inReplyTo":"ZScqLzGiDPZvLh2k@ugly","subject":"Re: [PATCH v8 1/3] unit tests: Add a project plan document","fromName":"Josh Steadmon","fromEmail":"steadmon@google.com","sentAt":"2023-11-01T17:31:48Z","receivedAt":"2023-11-01T17:31:58Z","isPatch":true,"sender":{"key":"steadmon@google.com","avatar":"https://avatars.githubusercontent.com/u/2654920?v=4"},"body":"On 2023.10.12 01:05, Oswald Buddenhagen wrote:\n> On Wed, Oct 11, 2023 at 02:14:03PM -0700, Josh Steadmon wrote:\n> > On 2023.10.10 10:57, Oswald Buddenhagen wrote:\n> > > On Mon, Oct 09, 2023 at 03:21:20PM -0700, Josh Steadmon wrote:\n> > > > +=== Comparison\n> > > > +\n> > > > +[format=\"csv\",options=\"header\",width=\"33%\"]\n> > > > +|=====\n> > > > +Framework,\"<>\",\"<>\",\"<>\",\"<>\",\"<>\",\"<>\",\"<>\",\"<>\",\"<>\",\"<>\",\"<>\",\"<>\"\n> > > > the redundancy seems unnecessary; asciidoc should automatically\n> > > use each\n> > > target's section title as the xreflabel.\n> > \n> > Hmm, this doesn't seem to work for me. It only renders as\n> > \"[anchor-label]\".\n> > \n> i thought\n> https://docs.asciidoctor.org/asciidoc/latest/attributes/id/#customize-automatic-xreftext\n> is pretty clear about it, though. maybe the actual tooling uses an older\n> version of the spec? or is buggy? or the placement of the titles is\n> incorrect? or this applies to different links or targets only? or am i\n> misreading something? or ...?\n> \n> regards\n\nI think the issue may be that asciidoc is the default formatter in\nDocumentation/Makefile, not asciidoctor.\n"},{"id":"484305","messageId":"ZUKPJsW1mT62Mcjy@google.com","threadId":"59937","inReplyTo":"CAP8UFD26X4MPbJs4KfNOgicLMb-wiuFZj3Hw17acMmmc_=vcqQ@mail.gmail.com","subject":"Re: [PATCH v8 1/3] unit tests: Add a project plan document","fromName":"Josh Steadmon","fromEmail":"steadmon@google.com","sentAt":"2023-11-01T17:47:18Z","receivedAt":"2023-11-01T17:47:25Z","isPatch":true,"sender":{"key":"steadmon@google.com","avatar":"https://avatars.githubusercontent.com/u/2654920?v=4"},"body":"On 2023.10.27 22:12, Christian Couder wrote:\n> On Tue, Oct 10, 2023 at 12:22 AM Josh Steadmon wrote:\n> >\n> > In our current testing environment, we spend a significant amount of\n> > effort crafting end-to-end tests for error conditions that could easily\n> > be captured by unit tests (or we simply forgo some hard-to-setup and\n> > rare error conditions). Describe what we hope to accomplish by\n> > implementing unit tests, and explain some open questions and milestones.\n> > Discuss desired features for test frameworks/harnesses, and provide a\n> > preliminary comparison of several different frameworks.\n> \n> Nit: Not sure why the test framework comparison is \"preliminary\" as we\n> have actually selected a unit test framework and are adding it in the\n> next patch of the series. I understand that this was perhaps written\n> before the choice was made, but maybe we might want to update that\n> now.\n\nFixed in v9, thanks.\n\n\n> > diff --git a/Documentation/technical/unit-tests.txt b/Documentation/technical/unit-tests.txt\n> > new file mode 100644\n> > index 0000000000..b7a89cc838\n> > --- /dev/null\n> > +++ b/Documentation/technical/unit-tests.txt\n> > @@ -0,0 +1,220 @@\n> > += Unit Testing\n> > +\n> > +In our current testing environment, we spend a significant amount of effort\n> > +crafting end-to-end tests for error conditions that could easily be captured by\n> > +unit tests (or we simply forgo some hard-to-setup and rare error conditions).\n> > +Unit tests additionally provide stability to the codebase and can simplify\n> > +debugging through isolation. Writing unit tests in pure C, rather than with our\n> > +current shell/test-tool helper setup, simplifies test setup, simplifies passing\n> > +data around (no shell-isms required), and reduces testing runtime by not\n> > +spawning a separate process for every test invocation.\n> > +\n> > +We believe that a large body of unit tests, living alongside the existing test\n> > +suite, will improve code quality for the Git project.\n> \n> I agree with that.\n> \n> > +== Choosing a framework\n> > +\n> > +We believe the best option is to implement a custom TAP framework for the Git\n> > +project. We use a version of the framework originally proposed in\n> > +https://lore.kernel.org/git/c902a166-98ce-afba-93f2-ea6027557176@gmail.com/[1].\n> \n> Nit: Logically I would think that our opinion should come after the\n> comparison and be backed by it.\n\nI intended this to be a quick summary for those who don't want to read\nthe whole doc. I clarified that and added a link to the selection\nrationale.\n\n\n> > +== Choosing a test harness\n> > +\n> > +During upstream discussion, it was occasionally noted that `prove` provides many\n> > +convenient features, such as scheduling slower tests first, or re-running\n> > +previously failed tests.\n> > +\n> > +While we already support the use of `prove` as a test harness for the shell\n> > +tests, it is not strictly required. The t/Makefile allows running shell tests\n> > +directly (though with interleaved output if parallelism is enabled). Git\n> > +developers who wish to use `prove` as a more advanced harness can do so by\n> > +setting DEFAULT_TEST_TARGET=prove in their config.mak.\n> > +\n> > +We will follow a similar approach for unit tests: by default the test\n> > +executables will be run directly from the t/Makefile, but `prove` can be\n> > +configured with DEFAULT_UNIT_TEST_TARGET=prove.\n> \n> Nice that it can be used.\n> \n> The rest of the file looks good.\n\nThanks for the review!\n"},{"id":"484306","messageId":"ZUKQ2eYmfMEhc_gH@google.com","threadId":"59937","inReplyTo":"20231016134421.21659-1-phillip.wood123@gmail.com","subject":"Re: [PATCH v8 2.5/3] fixup! unit tests: add TAP unit test framework","fromName":"Josh Steadmon","fromEmail":"steadmon@google.com","sentAt":"2023-11-01T17:54:33Z","receivedAt":"2023-11-01T17:54:40Z","isPatch":true,"sender":{"key":"steadmon@google.com","avatar":"https://avatars.githubusercontent.com/u/2654920?v=4"},"body":"On 2023.10.16 14:43, Phillip Wood wrote:\n> From: Phillip Wood \n> \n> Here are a couple of cleanups for the unit test framework that I\n> noticed.\n> \n> Update the documentation of the example custom check to reflect the\n> change in return value of test_assert() and mention that\n> checks should be careful when dereferencing pointer arguments.\n> \n> Also avoid evaluating macro augments twice in check_int() and\n> friends. The global variable test__tmp was introduced to avoid\n> evaluating the arguments to these macros more than once but the macros\n> failed to use it when passing the values being compared to\n> check_int_loc().\n> \n> Signed-off-by: Phillip Wood \n> ---\n> t/unit-tests/test-lib.h | 26 ++++++++++++++++----------\n> 1 file changed, 16 insertions(+), 10 deletions(-)\n\nApplied in v9, thanks!\n"},{"id":"484307","messageId":"ZUKQ8xtFSYbSBPXb@google.com","threadId":"59937","inReplyTo":"xmqq1qduo6yr.fsf@gitster.g","subject":"Re: [PATCH v8 2.5/3] fixup! unit tests: add TAP unit test framework","fromName":"Josh Steadmon","fromEmail":"steadmon@google.com","sentAt":"2023-11-01T17:54:59Z","receivedAt":"2023-11-01T17:55:06Z","isPatch":true,"sender":{"key":"steadmon@google.com","avatar":"https://avatars.githubusercontent.com/u/2654920?v=4"},"body":"On 2023.10.16 09:41, Junio C Hamano wrote:\n> Phillip Wood writes:\n> \n> > From: Phillip Wood \n> >\n> > Here are a couple of cleanups for the unit test framework that I\n> > noticed.\n> \n> Thanks. I trust that this will be squashed into the next update,\n> but in the meantime, I'll include it in the copy of the series I\n> have (without squashing). Here is another one I noticed.\n> \n> ----- >8 --------- >8 --------- >8 -----\n> Subject: [PATCH] fixup! ci: run unit tests in CI\n> \n> A CI job failed due to contrib/coccinelle/equals-null.cocci\n> and suggested this change, which seems sensible.\n> \n> Signed-off-by: Junio C Hamano \n> ---\n> t/unit-tests/t-strbuf.c | 2 +-\n> 1 file changed, 1 insertion(+), 1 deletion(-)\n\nApplied in v9, thanks!\n"},{"id":"484316","messageId":"ZULXMhD0ajESkde5@google.com","threadId":"59937","inReplyTo":"CAP8UFD3eY_i36YO0OcpAp9ey5KO0q-PrwvjSLRXKYQb=iZ8JCA@mail.gmail.com","subject":"Re: [PATCH v8 2/3] unit tests: add TAP unit test framework","fromName":"Josh Steadmon","fromEmail":"steadmon@google.com","sentAt":"2023-11-01T22:54:42Z","receivedAt":"2023-11-01T22:54:53Z","isPatch":true,"sender":{"key":"steadmon@google.com","avatar":"https://avatars.githubusercontent.com/u/2654920?v=4"},"body":"On 2023.10.27 22:15, Christian Couder wrote:\n> On Tue, Oct 10, 2023 at 12:22 AM Josh Steadmon wrote:\n> >\n> > From: Phillip Wood \n> >\n> > This patch contains an implementation for writing unit tests with TAP\n> > output. Each test is a function that contains one or more checks. The\n> > test is run with the TEST() macro and if any of the checks fail then the\n> > test will fail. A complete program that tests STRBUF_INIT would look\n> > like\n> >\n> > #include \"test-lib.h\"\n> > #include \"strbuf.h\"\n> >\n> > static void t_static_init(void)\n> > {\n> > struct strbuf buf = STRBUF_INIT;\n> >\n> > check_uint(buf.len, ==, 0);\n> > check_uint(buf.alloc, ==, 0);\n> > check_char(buf.buf[0], ==, '\\0');\n> > }\n> >\n> > int main(void)\n> > {\n> > TEST(t_static_init(), \"static initialization works);\n> >\n> > return test_done();\n> > }\n> >\n> > The output of this program would be\n> >\n> > ok 1 - static initialization works\n> > 1..1\n> >\n> > If any of the checks in a test fail then they print a diagnostic message\n> > to aid debugging and the test will be reported as failing. For example a\n> > failing integer check would look like\n> >\n> > # check \"x >= 3\" failed at my-test.c:102\n> \n> I wonder if it would be a bit better to say that the test was an\n> integer test for example with \"check_int(x >= 3) failed ...\"\n> \n> > # left: 2\n> > # right: 3\n> \n> I like \"expected\" and \"actual\" better than \"left\" and \"right\", not\n> sure how it's possible to have that in a way consistent with the shell\n> tests though.\n\nI also prefer expected/actual, but I don't think it's possible where we\naccept arbitrary operators, and I don't want to plumb a flag through to\nspecify whether to display left/right vs expected/actual.\n\n\n> > not ok 1 - x is greater than or equal to three\n> >\n> > There are a number of check functions implemented so far. check() checks\n> > a boolean condition, check_int(), check_uint() and check_char() take two\n> > values to compare and a comparison operator. check_str() will check if\n> > two strings are equal. Custom checks are simple to implement as shown in\n> > the comments above test_assert() in test-lib.h.\n> \n> Yeah, nice.\n> \n> > Tests can be skipped with test_skip() which can be supplied with a\n> > reason for skipping which it will print. Tests can print diagnostic\n> > messages with test_msg(). Checks that are known to fail can be wrapped\n> > in TEST_TODO().\n> \n> Maybe TEST_TOFIX() would be a bit more clear, but \"TODO\" is something\n> that is more likely to be searched for than \"TOFIX\", so Ok.\n> \n> > There are a couple of example test programs included in this\n> > patch. t-basic.c implements some self-tests and demonstrates the\n> > diagnostic output for failing test. The output of this program is\n> > checked by t0080-unit-test-output.sh. t-strbuf.c shows some example\n> > unit tests for strbuf.c\n> >\n> > The unit tests will be built as part of the default \"make all\" target,\n> > to avoid bitrot. If you wish to build just the unit tests, you can run\n> > \"make build-unit-tests\". To run the tests, you can use \"make unit-tests\"\n> > or run the test binaries directly, as in \"./t/unit-tests/bin/t-strbuf\".\n> \n> Nice!\n> \n> > +unit-tests-prove:\n> > + @echo \"*** prove - unit tests ***\"; $(PROVE) $(GIT_PROVE_OPTS) $(UNIT_TESTS)\n> \n> Nice, but DEFAULT_TEST_TARGET=prove isn't used. So not sure how\n> important or relevant the 'prove' related sections are in the\n> Documentation/technical/unit-tests.txt file introduced by the previous\n> patch.\n\nThe \"unit-tests\" target runs DEFAULT_UNIT_TEST_TARGET, which can be\noverridden to \"unit-tests-prove\".\n\n\n> > +int test_assert(const char *location, const char *check, int ok)\n> > +{\n> > + assert(ctx.running);\n> > +\n> > + if (ctx.result == RESULT_SKIP) {\n> > + test_msg(\"skipping check '%s' at %s\", check, location);\n> > + return 1;\n> > + } else if (!ctx.todo) {\n> \n> I think it would be a bit clearer without the \"else\" above and with\n> the \"if (!ctx.todo) {\" starting on a new line.\n\nFixed in v9.\n\n\n> > + if (ok) {\n> > + test_pass();\n> > + } else {\n> > + test_msg(\"check \\\"%s\\\" failed at %s\", check, location);\n> > + test_fail();\n> > + }\n> > + }\n> > +\n> > + return !!ok;\n> > +}\n> \n> Otherwise it looks good to me.\n\nThanks for the review!\n"},{"id":"484318","messageId":"ZULaty2ce5rUy8vz@google.com","threadId":"59937","inReplyTo":"93a18989-bf05-4318-8d85-cf23c0f32170@gmail.com","subject":"Re: [PATCH v8 0/3] Add unit test framework and project plan","fromName":"Josh Steadmon","fromEmail":"steadmon@google.com","sentAt":"2023-11-01T23:09:43Z","receivedAt":"2023-11-01T23:09:53Z","isPatch":true,"sender":{"key":"steadmon@google.com","avatar":"https://avatars.githubusercontent.com/u/2654920?v=4"},"body":"On 2023.10.16 11:07, phillip.wood123@gmail.com wrote:\n> Hi Josh\n> \n> Thanks for the update\n> \n> On 09/10/2023 23:21, Josh Steadmon wrote:\n> > In addition to reviewing the patches in this series, reviewers can help\n> > this series progress by chiming in on these remaining TODOs:\n> > - Figure out if we should split t-basic.c into multiple meta-tests, to\n> > avoid merge conflicts and changes to expected text in\n> > t0080-unit-test-output.sh.\n> \n> I think it depends on how many new tests we think we're going to want to add\n> here. I can see us adding a few more check_* macros (comparing object ids\n> and arrays of bytes spring to mind) and wanting to test them here, but\n> (perhaps naïvely) I don't expect huge amounts of churn here.\n\nThis is my feeling as well.\n\n\n> > - Figure out if we should de-duplicate assertions in t-strbuf.c at the\n> > cost of making tests less self-contained and diagnostic output less\n> > helpful.\n> \n> In principle we could pass the location information along to any helper\n> function, I'm not sure how easy that is at the moment. We can get reasonable\n> error messages by using the check*() macros in the helper and wrapping the\n> call to the helper with check() as well. For example\n> \n> static int assert_sane_strbuf(struct strbuf *buf)\n> {\n> \t/* Initialized strbufs should always have a non-NULL buffer */\n> \tif (!check(!!buf->buf))\n> \t\treturn 0;\n> \t/* Buffers should always be NUL-terminated */\n> \tif (!check_char(buf->buf[buf->len], ==, '\\0'))\n> \t\treturn 0;\n> \t/*\n> \t * Freshly-initialized strbufs may not have a dynamically allocated\n> \t * buffer\n> \t */\n> \tif (buf->len == 0 && buf->alloc == 0)\n> \t\treturn 1;\n> \t/* alloc must be at least one byte larger than len */\n> \treturn check_uint(buf->len, <, buf->alloc);\n> }\n> \n> and in the test function call it as\n> \n> \tcheck(assert_sane_strbuf(buf));\n> \n> which gives error messages like\n> \n> # check \"buf->len < buf->alloc\" failed at t/unit-tests/t-strbuf.c:43\n> # left: 5\n> # right: 0\n> # check \"assert_sane_strbuf(&buf)\" failed at t/unit-tests/t-strbuf.c:60\n> \n> So we can see where assert_sane_strbuf() was called and which assertion in\n> assert_sane_strbuf() failed.\n\nI like this approach. We'll need to document unit-test best practices,\nbut I think now that I'll want to do that in a separate series after\nthis one lands.\n\n\n> > - Figure out if we should collect unit tests statistics similar to the\n> > \"counts\" files for shell tests\n> \n> Unless someone has an immediate need for that I'd be tempted to leave it\n> wait until someone requests that data.\n> \n> > - Decide if it's OK to wait on sharding unit tests across \"sliced\" CI\n> > instances\n> \n> Hopefully the unit tests will run fast enough that we don't need to worry\n> about that in the early stages.\n> \n> > - Provide guidelines for writing new unit tests\n> \n> This is not a comprehensive list but we should recommend that\n> \n> - tests avoid leaking resources so the leak sanitizer see if the code\n> being tested has a resource leak.\n> \n> - tests check that pointers are not NULL before deferencing them to\n> avoid the whole program being taken down with SIGSEGV.\n> \n> - tests are written with easy debugging in mind - i.e. good diagnostic\n> messages. Hopefully the check* macros make that easy to do.\n\nThanks for the suggestions! I will make sure these make it into the best\npractices doc.\n\n\n> > Changes in v8:\n> > - Flipped return values for TEST, TEST_TODO, and check_* macros &\n> > functions. This makes it easier to reason about control flow for\n> > patterns like:\n> > if (check(some_condition)) { ... } > - Moved unit test binaries to t/unit-tests/bin to simplify .gitignore\n> > patterns.\n> \n> Thanks for the updates to the test library, the range diff looks good to me.\n> \n> > - Removed testing of some strbuf implementation details in t-strbuf.c\n> \n> I agree that makes sense. I think it would be good to update\n> assert_sane_strbuf() to use the check* macros as suggest above.\n\nFixed in v9.\n\n> Best Wishes\n> \n> Phillip\n"},{"id":"484320","messageId":"cover.1698881249.git.steadmon@google.com","threadId":"59937","inReplyTo":"0169ce6fb9ccafc089b74ae406db0d1a8ff8ac65.1688165272.git.steadmon@google.com","subject":"[PATCH v9 0/3] Add unit test framework and project plan","fromName":"Josh Steadmon","fromEmail":"steadmon@google.com","sentAt":"2023-11-01T23:31:14Z","receivedAt":"2023-11-01T23:31:28Z","isPatch":true,"sender":{"key":"steadmon@google.com","avatar":"https://avatars.githubusercontent.com/u/2654920?v=4"},"body":"In our current testing environment, we spend a significant amount of\neffort crafting end-to-end tests for error conditions that could easily\nbe captured by unit tests (or we simply forgo some hard-to-setup and\nrare error conditions). Unit tests additionally provide stability to the\ncodebase and can simplify debugging through isolation. Turning parts of\nGit into libraries[1] gives us the ability to run unit tests on the\nlibraries and to write unit tests in C. Writing unit tests in pure C,\nrather than with our current shell/test-tool helper setup, simplifies\ntest setup, simplifies passing data around (no shell-isms required), and\nreduces testing runtime by not spawning a separate process for every\ntest invocation.\n\nThis series begins with a project document covering our goals for adding\nunit tests and a discussion of alternative frameworks considered, as\nwell as the features used to evaluate them. A rendered preview of this\ndoc can be found at [2]. It also adds Phillip Wood's TAP implemenation\n(with some slightly re-worked Makefile rules) and a sample strbuf unit\ntest. Finally, we modify the configs for GitHub and Cirrus CI to run the\nunit tests. Sample runs showing successful CI runs can be found at [3],\n[4], and [5].\n\n[1] https://lore.kernel.org/git/CAJoAoZ=Cig_kLocxKGax31sU7Xe4==BGzC__Bg2_pr7krNq6MA@mail.gmail.com/\n[2] https://github.com/steadmon/git/blob/unit-tests-asciidoc/Documentation/technical/unit-tests.adoc\n[3] https://github.com/steadmon/git/actions/runs/5884659246/job/15959781385#step:4:1803\n[4] https://github.com/steadmon/git/actions/runs/5884659246/job/15959938401#step:5:186\n[5] https://cirrus-ci.com/task/6126304366428160 (unrelated tests failed,\n but note that t-strbuf ran successfully)\n\nChanges in v9:\n- Included some asciidoc cleanups suggested by Oswald Buddenhagen.\n- Applied a style fixup that Coccinelle complained about.\n- Applied some NULL-safety fixups.\n- Used check_*() more widely in t-strbuf helper functions\n\nChanges in v8:\n- Flipped return values for TEST, TEST_TODO, and check_* macros &\n functions. This makes it easier to reason about control flow for\n patterns like:\n if (check(some_condition)) { ... }\n- Moved unit test binaries to t/unit-tests/bin to simplify .gitignore\n patterns.\n- Removed testing of some strbuf implementation details in t-strbuf.c\n\nChanges in v7:\n- Fix corrupt diff in patch #2, sorry for the noise.\n\nChanges in v6:\n- Officially recommend using Phillip Wood's TAP framework\n- Add an example strbuf unit test using the TAP framework as well as\n Makefile integration\n- Run unit tests in CI\n\nChanges in v5:\n- Add comparison point \"License\".\n- Discuss feature priorities\n- Drop frameworks:\n - Incompatible licenses: libtap, cmocka\n - Missing source: MyTAP\n - No TAP support: µnit, cmockery, cmockery2, Unity, minunit, CUnit\n- Drop comparison point \"Coverage reports\": this can generally be\n handled by tools such as `gcov` regardless of the framework used.\n- Drop comparison point \"Inline tests\": there didn't seem to be\n strong interest from reviewers for this feature.\n- Drop comparison point \"Scheduling / re-running\": this was not\n supported by any of the main contenders, and is generally better\n handled by the harness rather than framework.\n- Drop comparison point \"Lazy test planning\": this was supported by\n all frameworks that provide TAP output.\n\nChanges in v4:\n- Add link anchors for the framework comparison dimensions\n- Explain \"Partial\" results for each dimension\n- Use consistent dimension names in the section headers and comparison\n tables\n- Add \"Project KLOC\", \"Adoption\", and \"Inline tests\" dimensions\n- Fill in a few of the missing entries in the comparison table\n\nChanges in v3:\n- Expand the doc with discussion of desired features and a WIP\n comparison.\n- Drop all implementation patches until a framework is selected.\n- Link to v2: https://lore.kernel.org/r/20230517-unit-tests-v2-v2-0-21b5b60f4b32@google.com\n\n\nJosh Steadmon (2):\n unit tests: Add a project plan document\n ci: run unit tests in CI\n\nPhillip Wood (1):\n unit tests: add TAP unit test framework\n\n .cirrus.yml | 2 +-\n Documentation/Makefile | 1 +\n Documentation/technical/unit-tests.txt | 240 ++++++++++++++++++\n Makefile | 28 ++-\n ci/run-build-and-tests.sh | 2 +\n ci/run-test-slice.sh | 5 +\n t/Makefile | 15 +-\n t/t0080-unit-test-output.sh | 58 +++++\n t/unit-tests/.gitignore | 1 +\n t/unit-tests/t-basic.c | 95 +++++++\n t/unit-tests/t-strbuf.c | 120 +++++++++\n t/unit-tests/test-lib.c | 329 +++++++++++++++++++++++++\n t/unit-tests/test-lib.h | 149 +++++++++++\n 13 files changed, 1040 insertions(+), 5 deletions(-)\n create mode 100644 Documentation/technical/unit-tests.txt\n create mode 100755 t/t0080-unit-test-output.sh\n create mode 100644 t/unit-tests/.gitignore\n create mode 100644 t/unit-tests/t-basic.c\n create mode 100644 t/unit-tests/t-strbuf.c\n create mode 100644 t/unit-tests/test-lib.c\n create mode 100644 t/unit-tests/test-lib.h\n\nRange-diff against v8:\n1: 81c5148a12 ! 1: f706ba9b68 unit tests: Add a project plan document\n @@ Commit message\n rare error conditions). Describe what we hope to accomplish by\n implementing unit tests, and explain some open questions and milestones.\n Discuss desired features for test frameworks/harnesses, and provide a\n - preliminary comparison of several different frameworks.\n + comparison of several different frameworks. Finally, document our\n + rationale for implementing a custom framework.\n \n Co-authored-by: Calvin Wan \n @@ Documentation/technical/unit-tests.txt (new)\n +can be made to work with a harness that we can choose later.\n +\n +\n -+== Choosing a framework\n ++== Summary\n +\n -+We believe the best option is to implement a custom TAP framework for the Git\n -+project. We use a version of the framework originally proposed in\n ++We believe the best way forward is to implement a custom TAP framework for the\n ++Git project. We use a version of the framework originally proposed in\n +https://lore.kernel.org/git/c902a166-98ce-afba-93f2-ea6027557176@gmail.com/[1].\n +\n ++See the <> section below for the\n ++rationale behind this decision.\n ++\n +\n +== Choosing a test harness\n +\n @@ Documentation/technical/unit-tests.txt (new)\n +configured with DEFAULT_UNIT_TEST_TARGET=prove.\n +\n +\n ++[[framework-selection]]\n +== Framework selection\n +\n +There are a variety of features we can use to rank the candidate frameworks, and\n @@ Documentation/technical/unit-tests.txt (new)\n +\n +=== Comparison\n +\n -+[format=\"csv\",options=\"header\",width=\"33%\"]\n ++:true: [lime-background]#True#\n ++:false: [red-background]#False#\n ++:partial: [yellow-background]#Partial#\n ++\n ++:gpl: [lime-background]#GPL v2#\n ++:isc: [lime-background]#ISC#\n ++:mit: [lime-background]#MIT#\n ++:expat: [lime-background]#Expat#\n ++:lgpl: [lime-background]#LGPL v2.1#\n ++\n ++:custom-impl: https://lore.kernel.org/git/c902a166-98ce-afba-93f2-ea6027557176@gmail.com/[Custom Git impl.]\n ++:greatest: https://github.com/silentbicycle/greatest[Greatest]\n ++:criterion: https://github.com/Snaipe/Criterion[Criterion]\n ++:c-tap: https://github.com/rra/c-tap-harness/[C TAP]\n ++:check: https://libcheck.github.io/check/[Check]\n ++\n ++[format=\"csv\",options=\"header\",width=\"33%\",subs=\"specialcharacters,attributes,quotes,macros\"]\n +|=====\n +Framework,\"<>\",\"<>\",\"<>\",\"<>\",\"<>\",\"<>\",\"<>\",\"<>\",\"<>\",\"<>\",\"<>\",\"<>\"\n -+https://lore.kernel.org/git/c902a166-98ce-afba-93f2-ea6027557176@gmail.com/[Custom Git impl.],[lime-background]#GPL v2#,[lime-background]#True#,[lime-background]#True#,[lime-background]#True#,[lime-background]#True#,[lime-background]#True#,[lime-background]#True#,[red-background]#False#,[red-background]#False#,[red-background]#False#,1,0\n -+https://github.com/silentbicycle/greatest[Greatest],[lime-background]#ISC#,[lime-background]#True#,[yellow-background]#Partial#,[lime-background]#True#,[yellow-background]#Partial#,[lime-background]#True#,[lime-background]#True#,[red-background]#False#,[red-background]#False#,[red-background]#False#,3,1400\n -+https://github.com/Snaipe/Criterion[Criterion],[lime-background]#MIT#,[red-background]#False#,[yellow-background]#Partial#,[lime-background]#True#,[lime-background]#True#,[lime-background]#True#,[lime-background]#True#,[lime-background]#True#,[red-background]#False#,[lime-background]#True#,19,1800\n -+https://github.com/rra/c-tap-harness/[C TAP],[lime-background]#Expat#,[lime-background]#True#,[yellow-background]#Partial#,[yellow-background]#Partial#,[lime-background]#True#,[red-background]#False#,[lime-background]#True#,[red-background]#False#,[red-background]#False#,[red-background]#False#,4,33\n -+https://libcheck.github.io/check/[Check],[lime-background]#LGPL v2.1#,[red-background]#False#,[yellow-background]#Partial#,[lime-background]#True#,[lime-background]#True#,[lime-background]#True#,[red-background]#False#,[red-background]#False#,[red-background]#False#,[lime-background]#True#,17,973\n ++{custom-impl},{gpl},{true},{true},{true},{true},{true},{true},{false},{false},{false},1,0\n ++{greatest},{isc},{true},{partial},{true},{partial},{true},{true},{false},{false},{false},3,1400\n ++{criterion},{mit},{false},{partial},{true},{true},{true},{true},{true},{false},{true},19,1800\n ++{c-tap},{expat},{true},{partial},{partial},{true},{false},{true},{false},{false},{false},4,33\n ++{check},{lgpl},{false},{partial},{true},{true},{true},{false},{false},{false},{true},17,973\n +|=====\n +\n +=== Additional framework candidates\n2: 00d3c95a81 ! 2: 8b831f4937 unit tests: add TAP unit test framework\n @@ t/unit-tests/t-strbuf.c (new)\n +static int assert_sane_strbuf(struct strbuf *buf)\n +{\n +\t/* Initialized strbufs should always have a non-NULL buffer */\n -+\tif (buf->buf == NULL)\n ++\tif (!check(!!buf->buf))\n +\t\treturn 0;\n +\t/* Buffers should always be NUL-terminated */\n -+\tif (buf->buf[buf->len] != '\\0')\n ++\tif (!check_char(buf->buf[buf->len], ==, '\\0'))\n +\t\treturn 0;\n +\t/*\n +\t * Freshly-initialized strbufs may not have a dynamically allocated\n @@ t/unit-tests/t-strbuf.c (new)\n +\tif (buf->len == 0 && buf->alloc == 0)\n +\t\treturn 1;\n +\t/* alloc must be at least one byte larger than len */\n -+\treturn buf->len + 1 <= buf->alloc;\n ++\treturn check_uint(buf->len, <, buf->alloc);\n +}\n +\n +static void t_static_init(void)\n @@ t/unit-tests/test-lib.h (new)\n +\n +/*\n + * Test checks are built around test_assert(). checks return 1 on\n -+ * success, 0 on failure. If any check fails then the test will\n -+ * fail. To create a custom check define a function that wraps\n -+ * test_assert() and a macro to wrap that function. For example:\n ++ * success, 0 on failure. If any check fails then the test will fail. To\n ++ * create a custom check define a function that wraps test_assert() and\n ++ * a macro to wrap that function to provide a source location and\n ++ * stringified arguments. Custom checks that take pointer arguments\n ++ * should be careful to check that they are non-NULL before\n ++ * dereferencing them. For example:\n + *\n + * static int check_oid_loc(const char *loc, const char *check,\n + *\t\t\t struct object_id *a, struct object_id *b)\n + * {\n -+ *\t int res = test_assert(loc, check, oideq(a, b));\n ++ *\t int res = test_assert(loc, check, a && b && oideq(a, b));\n + *\n -+ *\t if (res) {\n -+ *\t\t test_msg(\" left: %s\", oid_to_hex(a);\n -+ *\t\t test_msg(\" right: %s\", oid_to_hex(a);\n ++ *\t if (!res) {\n ++ *\t\t test_msg(\" left: %s\", a ? oid_to_hex(a) : \"NULL\";\n ++ *\t\t test_msg(\" right: %s\", b ? oid_to_hex(a) : \"NULL\";\n + *\n + *\t }\n + *\t return res;\n @@ t/unit-tests/test-lib.h (new)\n +#define check_int(a, op, b)\t\t\t\t\t\t\\\n +\t(test__tmp[0].i = (a), test__tmp[1].i = (b),\t\t\t\\\n +\t check_int_loc(TEST_LOCATION(), #a\" \"#op\" \"#b,\t\t\t\\\n -+\t\t test__tmp[0].i op test__tmp[1].i, a, b))\n ++\t\t test__tmp[0].i op test__tmp[1].i,\t\t\\\n ++\t\t test__tmp[0].i, test__tmp[1].i))\n +int check_int_loc(const char *loc, const char *check, int ok,\n +\t\t intmax_t a, intmax_t b);\n +\n @@ t/unit-tests/test-lib.h (new)\n +#define check_uint(a, op, b)\t\t\t\t\t\t\\\n +\t(test__tmp[0].u = (a), test__tmp[1].u = (b),\t\t\t\\\n +\t check_uint_loc(TEST_LOCATION(), #a\" \"#op\" \"#b,\t\t\t\\\n -+\t\t\ttest__tmp[0].u op test__tmp[1].u, a, b))\n ++\t\t\ttest__tmp[0].u op test__tmp[1].u,\t\t\\\n ++\t\t\ttest__tmp[0].u, test__tmp[1].u))\n +int check_uint_loc(const char *loc, const char *check, int ok,\n +\t\t uintmax_t a, uintmax_t b);\n +\n @@ t/unit-tests/test-lib.h (new)\n +#define check_char(a, op, b)\t\t\t\t\t\t\\\n +\t(test__tmp[0].c = (a), test__tmp[1].c = (b),\t\t\t\\\n +\t check_char_loc(TEST_LOCATION(), #a\" \"#op\" \"#b,\t\t\t\\\n -+\t\t\ttest__tmp[0].c op test__tmp[1].c, a, b))\n ++\t\t\ttest__tmp[0].c op test__tmp[1].c,\t\t\\\n ++\t\t\ttest__tmp[0].c, test__tmp[1].c))\n +int check_char_loc(const char *loc, const char *check, int ok,\n +\t\t char a, char b);\n +\n3: aa1dfa4892 = 3: 08d27bb5f9 ci: run unit tests in CI\n\nbase-commit: a9e066fa63149291a55f383cfa113d8bdbdaa6b3\n-- \n2.42.0.869.gea05f2083d-goog\n\n"},{"id":"484321","messageId":"f706ba9b682e7c4070d49086ad3af582bf269c79.1698881249.git.steadmon@google.com","threadId":"59937","inReplyTo":"cover.1698881249.git.steadmon@google.com","subject":"[PATCH v9 1/3] unit tests: Add a project plan document","fromName":"Josh Steadmon","fromEmail":"steadmon@google.com","sentAt":"2023-11-01T23:31:15Z","receivedAt":"2023-11-01T23:31:29Z","isPatch":true,"sender":{"key":"steadmon@google.com","avatar":"https://avatars.githubusercontent.com/u/2654920?v=4"},"body":"In our current testing environment, we spend a significant amount of\neffort crafting end-to-end tests for error conditions that could easily\nbe captured by unit tests (or we simply forgo some hard-to-setup and\nrare error conditions). Describe what we hope to accomplish by\nimplementing unit tests, and explain some open questions and milestones.\nDiscuss desired features for test frameworks/harnesses, and provide a\ncomparison of several different frameworks. Finally, document our\nrationale for implementing a custom framework.\n\nCo-authored-by: Calvin Wan \nSigned-off-by: Calvin Wan \nSigned-off-by: Josh Steadmon \n---\n Documentation/Makefile | 1 +\n Documentation/technical/unit-tests.txt | 240 +++++++++++++++++++++++++\n 2 files changed, 241 insertions(+)\n create mode 100644 Documentation/technical/unit-tests.txt\n\ndiff --git a/Documentation/Makefile b/Documentation/Makefile\nindex b629176d7d..3f2383a12c 100644\n--- a/Documentation/Makefile\n+++ b/Documentation/Makefile\n@@ -122,6 +122,7 @@ TECH_DOCS += technical/scalar\n TECH_DOCS += technical/send-pack-pipeline\n TECH_DOCS += technical/shallow\n TECH_DOCS += technical/trivial-merge\n+TECH_DOCS += technical/unit-tests\n SP_ARTICLES += $(TECH_DOCS)\n SP_ARTICLES += technical/api-index\n \ndiff --git a/Documentation/technical/unit-tests.txt b/Documentation/technical/unit-tests.txt\nnew file mode 100644\nindex 0000000000..206037ffb1\n--- /dev/null\n+++ b/Documentation/technical/unit-tests.txt\n@@ -0,0 +1,240 @@\n+= Unit Testing\n+\n+In our current testing environment, we spend a significant amount of effort\n+crafting end-to-end tests for error conditions that could easily be captured by\n+unit tests (or we simply forgo some hard-to-setup and rare error conditions).\n+Unit tests additionally provide stability to the codebase and can simplify\n+debugging through isolation. Writing unit tests in pure C, rather than with our\n+current shell/test-tool helper setup, simplifies test setup, simplifies passing\n+data around (no shell-isms required), and reduces testing runtime by not\n+spawning a separate process for every test invocation.\n+\n+We believe that a large body of unit tests, living alongside the existing test\n+suite, will improve code quality for the Git project.\n+\n+== Definitions\n+\n+For the purposes of this document, we'll use *test framework* to refer to\n+projects that support writing test cases and running tests within the context\n+of a single executable. *Test harness* will refer to projects that manage\n+running multiple executables (each of which may contain multiple test cases) and\n+aggregating their results.\n+\n+In reality, these terms are not strictly defined, and many of the projects\n+discussed below contain features from both categories.\n+\n+For now, we will evaluate projects solely on their framework features. Since we\n+are relying on having TAP output (see below), we can assume that any framework\n+can be made to work with a harness that we can choose later.\n+\n+\n+== Summary\n+\n+We believe the best way forward is to implement a custom TAP framework for the\n+Git project. We use a version of the framework originally proposed in\n+https://lore.kernel.org/git/c902a166-98ce-afba-93f2-ea6027557176@gmail.com/[1].\n+\n+See the <> section below for the\n+rationale behind this decision.\n+\n+\n+== Choosing a test harness\n+\n+During upstream discussion, it was occasionally noted that `prove` provides many\n+convenient features, such as scheduling slower tests first, or re-running\n+previously failed tests.\n+\n+While we already support the use of `prove` as a test harness for the shell\n+tests, it is not strictly required. The t/Makefile allows running shell tests\n+directly (though with interleaved output if parallelism is enabled). Git\n+developers who wish to use `prove` as a more advanced harness can do so by\n+setting DEFAULT_TEST_TARGET=prove in their config.mak.\n+\n+We will follow a similar approach for unit tests: by default the test\n+executables will be run directly from the t/Makefile, but `prove` can be\n+configured with DEFAULT_UNIT_TEST_TARGET=prove.\n+\n+\n+[[framework-selection]]\n+== Framework selection\n+\n+There are a variety of features we can use to rank the candidate frameworks, and\n+those features have different priorities:\n+\n+* Critical features: we probably won't consider a framework without these\n+** Can we legally / easily use the project?\n+*** <>\n+*** <>\n+*** <>\n+*** <>\n+** Does the project support our bare-minimum needs?\n+*** <>\n+*** <>\n+*** <>\n+* Nice-to-have features:\n+** <>\n+** <>\n+** <>\n+* Tie-breaker stats\n+** <>\n+** <>\n+\n+[[license]]\n+=== License\n+\n+We must be able to legally use the framework in connection with Git. As Git is\n+licensed only under GPLv2, we must eliminate any LGPLv3, GPLv3, or Apache 2.0\n+projects.\n+\n+[[vendorable-or-ubiquitous]]\n+=== Vendorable or ubiquitous\n+\n+We want to avoid forcing Git developers to install new tools just to run unit\n+tests. Any prospective frameworks and harnesses must either be vendorable\n+(meaning, we can copy their source directly into Git's repository), or so\n+ubiquitous that it is reasonable to expect that most developers will have the\n+tools installed already.\n+\n+[[maintainable-extensible]]\n+=== Maintainable / extensible\n+\n+It is unlikely that any pre-existing project perfectly fits our needs, so any\n+project we select will need to be actively maintained and open to accepting\n+changes. Alternatively, assuming we are vendoring the source into our repo, it\n+must be simple enough that Git developers can feel comfortable making changes as\n+needed to our version.\n+\n+In the comparison table below, \"True\" means that the framework seems to have\n+active developers, that it is simple enough that Git developers can make changes\n+to it, and that the project seems open to accepting external contributions (or\n+that it is vendorable). \"Partial\" means that at least one of the above\n+conditions holds.\n+\n+[[major-platform-support]]\n+=== Major platform support\n+\n+At a bare minimum, unit-testing must work on Linux, MacOS, and Windows.\n+\n+In the comparison table below, \"True\" means that it works on all three major\n+platforms with no issues. \"Partial\" means that there may be annoyances on one or\n+more platforms, but it is still usable in principle.\n+\n+[[tap-support]]\n+=== TAP support\n+\n+The https://testanything.org/[Test Anything Protocol] is a text-based interface\n+that allows tests to communicate with a test harness. It is already used by\n+Git's integration test suite. Supporting TAP output is a mandatory feature for\n+any prospective test framework.\n+\n+In the comparison table below, \"True\" means this is natively supported.\n+\"Partial\" means TAP output must be generated by post-processing the native\n+output.\n+\n+Frameworks that do not have at least Partial support will not be evaluated\n+further.\n+\n+[[diagnostic-output]]\n+=== Diagnostic output\n+\n+When a test case fails, the framework must generate enough diagnostic output to\n+help developers find the appropriate test case in source code in order to debug\n+the failure.\n+\n+[[runtime-skippable-tests]]\n+=== Runtime-skippable tests\n+\n+Test authors may wish to skip certain test cases based on runtime circumstances,\n+so the framework should support this.\n+\n+[[parallel-execution]]\n+=== Parallel execution\n+\n+Ideally, we will build up a significant collection of unit test cases, most\n+likely split across multiple executables. It will be necessary to run these\n+tests in parallel to enable fast develop-test-debug cycles.\n+\n+In the comparison table below, \"True\" means that individual test cases within a\n+single test executable can be run in parallel. We assume that executable-level\n+parallelism can be handled by the test harness.\n+\n+[[mock-support]]\n+=== Mock support\n+\n+Unit test authors may wish to test code that interacts with objects that may be\n+inconvenient to handle in a test (e.g. interacting with a network service).\n+Mocking allows test authors to provide a fake implementation of these objects\n+for more convenient tests.\n+\n+[[signal-error-handling]]\n+=== Signal & error handling\n+\n+The test framework should fail gracefully when test cases are themselves buggy\n+or when they are interrupted by signals during runtime.\n+\n+[[project-kloc]]\n+=== Project KLOC\n+\n+The size of the project, in thousands of lines of code as measured by\n+https://dwheeler.com/sloccount/[sloccount] (rounded up to the next multiple of\n+1,000). As a tie-breaker, we probably prefer a project with fewer LOC.\n+\n+[[adoption]]\n+=== Adoption\n+\n+As a tie-breaker, we prefer a more widely-used project. We use the number of\n+GitHub / GitLab stars to estimate this.\n+\n+\n+=== Comparison\n+\n+:true: [lime-background]#True#\n+:false: [red-background]#False#\n+:partial: [yellow-background]#Partial#\n+\n+:gpl: [lime-background]#GPL v2#\n+:isc: [lime-background]#ISC#\n+:mit: [lime-background]#MIT#\n+:expat: [lime-background]#Expat#\n+:lgpl: [lime-background]#LGPL v2.1#\n+\n+:custom-impl: https://lore.kernel.org/git/c902a166-98ce-afba-93f2-ea6027557176@gmail.com/[Custom Git impl.]\n+:greatest: https://github.com/silentbicycle/greatest[Greatest]\n+:criterion: https://github.com/Snaipe/Criterion[Criterion]\n+:c-tap: https://github.com/rra/c-tap-harness/[C TAP]\n+:check: https://libcheck.github.io/check/[Check]\n+\n+[format=\"csv\",options=\"header\",width=\"33%\",subs=\"specialcharacters,attributes,quotes,macros\"]\n+|=====\n+Framework,\"<>\",\"<>\",\"<>\",\"<>\",\"<>\",\"<>\",\"<>\",\"<>\",\"<>\",\"<>\",\"<>\",\"<>\"\n+{custom-impl},{gpl},{true},{true},{true},{true},{true},{true},{false},{false},{false},1,0\n+{greatest},{isc},{true},{partial},{true},{partial},{true},{true},{false},{false},{false},3,1400\n+{criterion},{mit},{false},{partial},{true},{true},{true},{true},{true},{false},{true},19,1800\n+{c-tap},{expat},{true},{partial},{partial},{true},{false},{true},{false},{false},{false},4,33\n+{check},{lgpl},{false},{partial},{true},{true},{true},{false},{false},{false},{true},17,973\n+|=====\n+\n+=== Additional framework candidates\n+\n+Several suggested frameworks have been eliminated from consideration:\n+\n+* Incompatible licenses:\n+** https://github.com/zorgnax/libtap[libtap] (LGPL v3)\n+** https://cmocka.org/[cmocka] (Apache 2.0)\n+* Missing source: https://www.kindahl.net/mytap/doc/index.html[MyTap]\n+* No TAP support:\n+** https://nemequ.github.io/munit/[µnit]\n+** https://github.com/google/cmockery[cmockery]\n+** https://github.com/lpabon/cmockery2[cmockery2]\n+** https://github.com/ThrowTheSwitch/Unity[Unity]\n+** https://github.com/siu/minunit[minunit]\n+** https://cunit.sourceforge.net/[CUnit]\n+\n+\n+== Milestones\n+\n+* Add useful tests of library-like code\n+* Integrate with\n+ https://lore.kernel.org/git/20230502211454.1673000-1-calvinwan@google.com/[stdlib\n+ work]\n+* Run alongside regular `make test` target\n-- \n2.42.0.869.gea05f2083d-goog\n\n"},{"id":"484322","messageId":"08d27bb5f9b9c9b92f2ab75a4caf37fe49ac89f9.1698881249.git.steadmon@google.com","threadId":"59937","inReplyTo":"cover.1698881249.git.steadmon@google.com","subject":"[PATCH v9 3/3] ci: run unit tests in CI","fromName":"Josh Steadmon","fromEmail":"steadmon@google.com","sentAt":"2023-11-01T23:31:17Z","receivedAt":"2023-11-01T23:31:33Z","isPatch":true,"sender":{"key":"steadmon@google.com","avatar":"https://avatars.githubusercontent.com/u/2654920?v=4"},"body":"Run unit tests in both Cirrus and GitHub CI. For sharded CI instances\n(currently just Windows on GitHub), run only on the first shard. This is\nOK while we have only a single unit test executable, but we may wish to\ndistribute tests more evenly when we add new unit tests in the future.\n\nWe may also want to add more status output in our unit test framework,\nso that we can do similar post-processing as in\nci/lib.sh:handle_failed_tests().\n\nSigned-off-by: Josh Steadmon \n---\n .cirrus.yml | 2 +-\n ci/run-build-and-tests.sh | 2 ++\n ci/run-test-slice.sh | 5 +++++\n 3 files changed, 8 insertions(+), 1 deletion(-)\n\ndiff --git a/.cirrus.yml b/.cirrus.yml\nindex 4860bebd32..b6280692d2 100644\n--- a/.cirrus.yml\n+++ b/.cirrus.yml\n@@ -19,4 +19,4 @@ freebsd_12_task:\n build_script:\n - su git -c gmake\n test_script:\n- - su git -c 'gmake test'\n+ - su git -c 'gmake DEFAULT_UNIT_TEST_TARGET=unit-tests-prove test unit-tests'\ndiff --git a/ci/run-build-and-tests.sh b/ci/run-build-and-tests.sh\nindex 2528f25e31..7a1466b868 100755\n--- a/ci/run-build-and-tests.sh\n+++ b/ci/run-build-and-tests.sh\n@@ -50,6 +50,8 @@ if test -n \"$run_tests\"\n then\n \tgroup \"Run tests\" make test ||\n \thandle_failed_tests\n+\tgroup \"Run unit tests\" \\\n+\t\tmake DEFAULT_UNIT_TEST_TARGET=unit-tests-prove unit-tests\n fi\n check_unignored_build_artifacts\n \ndiff --git a/ci/run-test-slice.sh b/ci/run-test-slice.sh\nindex a3c67956a8..ae8094382f 100755\n--- a/ci/run-test-slice.sh\n+++ b/ci/run-test-slice.sh\n@@ -15,4 +15,9 @@ group \"Run tests\" make --quiet -C t T=\"$(cd t &&\n \ttr '\\n' ' ')\" ||\n handle_failed_tests\n \n+# We only have one unit test at the moment, so run it in the first slice\n+if [ \"$1\" == \"0\" ] ; then\n+\tgroup \"Run unit tests\" make --quiet -C t unit-tests-prove\n+fi\n+\n check_unignored_build_artifacts\n-- \n2.42.0.869.gea05f2083d-goog\n\n"},{"id":"484323","messageId":"8b831f4937a167519b77c724fcbc36884bbfe43d.1698881249.git.steadmon@google.com","threadId":"59937","inReplyTo":"cover.1698881249.git.steadmon@google.com","subject":"[PATCH v9 2/3] unit tests: add TAP unit test framework","fromName":"Josh Steadmon","fromEmail":"steadmon@google.com","sentAt":"2023-11-01T23:31:16Z","receivedAt":"2023-11-01T23:31:33Z","isPatch":true,"sender":{"key":"steadmon@google.com","avatar":"https://avatars.githubusercontent.com/u/2654920?v=4"},"body":"From: Phillip Wood \n\nThis patch contains an implementation for writing unit tests with TAP\noutput. Each test is a function that contains one or more checks. The\ntest is run with the TEST() macro and if any of the checks fail then the\ntest will fail. A complete program that tests STRBUF_INIT would look\nlike\n\n #include \"test-lib.h\"\n #include \"strbuf.h\"\n\n static void t_static_init(void)\n {\n struct strbuf buf = STRBUF_INIT;\n\n check_uint(buf.len, ==, 0);\n check_uint(buf.alloc, ==, 0);\n check_char(buf.buf[0], ==, '\\0');\n }\n\n int main(void)\n {\n TEST(t_static_init(), \"static initialization works);\n\n return test_done();\n }\n\nThe output of this program would be\n\n ok 1 - static initialization works\n 1..1\n\nIf any of the checks in a test fail then they print a diagnostic message\nto aid debugging and the test will be reported as failing. For example a\nfailing integer check would look like\n\n # check \"x >= 3\" failed at my-test.c:102\n # left: 2\n # right: 3\n not ok 1 - x is greater than or equal to three\n\nThere are a number of check functions implemented so far. check() checks\na boolean condition, check_int(), check_uint() and check_char() take two\nvalues to compare and a comparison operator. check_str() will check if\ntwo strings are equal. Custom checks are simple to implement as shown in\nthe comments above test_assert() in test-lib.h.\n\nTests can be skipped with test_skip() which can be supplied with a\nreason for skipping which it will print. Tests can print diagnostic\nmessages with test_msg(). Checks that are known to fail can be wrapped\nin TEST_TODO().\n\nThere are a couple of example test programs included in this\npatch. t-basic.c implements some self-tests and demonstrates the\ndiagnostic output for failing test. The output of this program is\nchecked by t0080-unit-test-output.sh. t-strbuf.c shows some example\nunit tests for strbuf.c\n\nThe unit tests will be built as part of the default \"make all\" target,\nto avoid bitrot. If you wish to build just the unit tests, you can run\n\"make build-unit-tests\". To run the tests, you can use \"make unit-tests\"\nor run the test binaries directly, as in \"./t/unit-tests/bin/t-strbuf\".\n\nSigned-off-by: Phillip Wood \nSigned-off-by: Josh Steadmon \n---\n Makefile | 28 ++-\n t/Makefile | 15 +-\n t/t0080-unit-test-output.sh | 58 +++++++\n t/unit-tests/.gitignore | 1 +\n t/unit-tests/t-basic.c | 95 +++++++++++\n t/unit-tests/t-strbuf.c | 120 +++++++++++++\n t/unit-tests/test-lib.c | 329 ++++++++++++++++++++++++++++++++++++\n t/unit-tests/test-lib.h | 149 ++++++++++++++++\n 8 files changed, 791 insertions(+), 4 deletions(-)\n create mode 100755 t/t0080-unit-test-output.sh\n create mode 100644 t/unit-tests/.gitignore\n create mode 100644 t/unit-tests/t-basic.c\n create mode 100644 t/unit-tests/t-strbuf.c\n create mode 100644 t/unit-tests/test-lib.c\n create mode 100644 t/unit-tests/test-lib.h\n\ndiff --git a/Makefile b/Makefile\nindex e440728c24..18c13f06c0 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -682,6 +682,9 @@ TEST_BUILTINS_OBJS =\n TEST_OBJS =\n TEST_PROGRAMS_NEED_X =\n THIRD_PARTY_SOURCES =\n+UNIT_TEST_PROGRAMS =\n+UNIT_TEST_DIR = t/unit-tests\n+UNIT_TEST_BIN = $(UNIT_TEST_DIR)/bin\n \n # Having this variable in your environment would break pipelines because\n # you cause \"cd\" to echo its destination to stdout. It can also take\n@@ -1331,6 +1334,12 @@ THIRD_PARTY_SOURCES += compat/regex/%\n THIRD_PARTY_SOURCES += sha1collisiondetection/%\n THIRD_PARTY_SOURCES += sha1dc/%\n \n+UNIT_TEST_PROGRAMS += t-basic\n+UNIT_TEST_PROGRAMS += t-strbuf\n+UNIT_TEST_PROGS = $(patsubst %,$(UNIT_TEST_BIN)/%$X,$(UNIT_TEST_PROGRAMS))\n+UNIT_TEST_OBJS = $(patsubst %,$(UNIT_TEST_DIR)/%.o,$(UNIT_TEST_PROGRAMS))\n+UNIT_TEST_OBJS += $(UNIT_TEST_DIR)/test-lib.o\n+\n # xdiff and reftable libs may in turn depend on what is in libgit.a\n GITLIBS = common-main.o $(LIB_FILE) $(XDIFF_LIB) $(REFTABLE_LIB) $(LIB_FILE)\n EXTLIBS =\n@@ -2672,6 +2681,7 @@ OBJECTS += $(TEST_OBJS)\n OBJECTS += $(XDIFF_OBJS)\n OBJECTS += $(FUZZ_OBJS)\n OBJECTS += $(REFTABLE_OBJS) $(REFTABLE_TEST_OBJS)\n+OBJECTS += $(UNIT_TEST_OBJS)\n \n ifndef NO_CURL\n \tOBJECTS += http.o http-walker.o remote-curl.o\n@@ -3167,7 +3177,7 @@ endif\n \n test_bindir_programs := $(patsubst %,bin-wrappers/%,$(BINDIR_PROGRAMS_NEED_X) $(BINDIR_PROGRAMS_NO_X) $(TEST_PROGRAMS_NEED_X))\n \n-all:: $(TEST_PROGRAMS) $(test_bindir_programs)\n+all:: $(TEST_PROGRAMS) $(test_bindir_programs) $(UNIT_TEST_PROGS)\n \n bin-wrappers/%: wrap-for-bin.sh\n \t$(call mkdir_p_parent_template)\n@@ -3592,7 +3602,7 @@ endif\n \n artifacts-tar:: $(ALL_COMMANDS_TO_INSTALL) $(SCRIPT_LIB) $(OTHER_PROGRAMS) \\\n \t\tGIT-BUILD-OPTIONS $(TEST_PROGRAMS) $(test_bindir_programs) \\\n-\t\t$(MOFILES)\n+\t\t$(UNIT_TEST_PROGS) $(MOFILES)\n \t$(QUIET_SUBDIR0)templates $(QUIET_SUBDIR1) \\\n \t\tSHELL_PATH='$(SHELL_PATH_SQ)' PERL_PATH='$(PERL_PATH_SQ)'\n \ttest -n \"$(ARTIFACTS_DIRECTORY)\"\n@@ -3653,7 +3663,7 @@ clean: profile-clean coverage-clean cocciclean\n \t$(RM) $(OBJECTS)\n \t$(RM) $(LIB_FILE) $(XDIFF_LIB) $(REFTABLE_LIB) $(REFTABLE_TEST_LIB)\n \t$(RM) $(ALL_PROGRAMS) $(SCRIPT_LIB) $(BUILT_INS) $(OTHER_PROGRAMS)\n-\t$(RM) $(TEST_PROGRAMS)\n+\t$(RM) $(TEST_PROGRAMS) $(UNIT_TEST_PROGS)\n \t$(RM) $(FUZZ_PROGRAMS)\n \t$(RM) $(SP_OBJ)\n \t$(RM) $(HCC)\n@@ -3831,3 +3841,15 @@ $(FUZZ_PROGRAMS): all\n \t\t$(XDIFF_OBJS) $(EXTLIBS) git.o $@.o $(LIB_FUZZING_ENGINE) -o $@\n \n fuzz-all: $(FUZZ_PROGRAMS)\n+\n+$(UNIT_TEST_BIN):\n+\t@mkdir -p $(UNIT_TEST_BIN)\n+\n+$(UNIT_TEST_PROGS): $(UNIT_TEST_BIN)/%$X: $(UNIT_TEST_DIR)/%.o $(UNIT_TEST_DIR)/test-lib.o $(GITLIBS) GIT-LDFLAGS $(UNIT_TEST_BIN)\n+\t$(QUIET_LINK)$(CC) $(ALL_CFLAGS) -o $@ $(ALL_LDFLAGS) \\\n+\t\t$(filter %.o,$^) $(filter %.a,$^) $(LIBS)\n+\n+.PHONY: build-unit-tests unit-tests\n+build-unit-tests: $(UNIT_TEST_PROGS)\n+unit-tests: $(UNIT_TEST_PROGS)\n+\t$(MAKE) -C t/ unit-tests\ndiff --git a/t/Makefile b/t/Makefile\nindex 3e00cdd801..75d9330437 100644\n--- a/t/Makefile\n+++ b/t/Makefile\n@@ -17,6 +17,7 @@ TAR ?= $(TAR)\n RM ?= rm -f\n PROVE ?= prove\n DEFAULT_TEST_TARGET ?= test\n+DEFAULT_UNIT_TEST_TARGET ?= unit-tests-raw\n TEST_LINT ?= test-lint\n \n ifdef TEST_OUTPUT_DIRECTORY\n@@ -41,6 +42,7 @@ TPERF = $(sort $(wildcard perf/p[0-9][0-9][0-9][0-9]-*.sh))\n TINTEROP = $(sort $(wildcard interop/i[0-9][0-9][0-9][0-9]-*.sh))\n CHAINLINTTESTS = $(sort $(patsubst chainlint/%.test,%,$(wildcard chainlint/*.test)))\n CHAINLINT = '$(PERL_PATH_SQ)' chainlint.pl\n+UNIT_TESTS = $(sort $(filter-out unit-tests/bin/t-basic%,$(wildcard unit-tests/bin/t-*)))\n \n # `test-chainlint` (which is a dependency of `test-lint`, `test` and `prove`)\n # checks all tests in all scripts via a single invocation, so tell individual\n@@ -65,6 +67,17 @@ prove: pre-clean check-chainlint $(TEST_LINT)\n $(T):\n \t@echo \"*** $@ ***\"; '$(TEST_SHELL_PATH_SQ)' $@ $(GIT_TEST_OPTS)\n \n+$(UNIT_TESTS):\n+\t@echo \"*** $@ ***\"; $@\n+\n+.PHONY: unit-tests unit-tests-raw unit-tests-prove\n+unit-tests: $(DEFAULT_UNIT_TEST_TARGET)\n+\n+unit-tests-raw: $(UNIT_TESTS)\n+\n+unit-tests-prove:\n+\t@echo \"*** prove - unit tests ***\"; $(PROVE) $(GIT_PROVE_OPTS) $(UNIT_TESTS)\n+\n pre-clean:\n \t$(RM) -r '$(TEST_RESULTS_DIRECTORY_SQ)'\n \n@@ -149,4 +162,4 @@ perf:\n \t$(MAKE) -C perf/ all\n \n .PHONY: pre-clean $(T) aggregate-results clean valgrind perf \\\n-\tcheck-chainlint clean-chainlint test-chainlint\n+\tcheck-chainlint clean-chainlint test-chainlint $(UNIT_TESTS)\ndiff --git a/t/t0080-unit-test-output.sh b/t/t0080-unit-test-output.sh\nnew file mode 100755\nindex 0000000000..961b54b06c\n--- /dev/null\n+++ b/t/t0080-unit-test-output.sh\n@@ -0,0 +1,58 @@\n+#!/bin/sh\n+\n+test_description='Test the output of the unit test framework'\n+\n+. ./test-lib.sh\n+\n+test_expect_success 'TAP output from unit tests' '\n+\tcat >expect <<-EOF &&\n+\tok 1 - passing test\n+\tok 2 - passing test and assertion return 1\n+\t# check \"1 == 2\" failed at t/unit-tests/t-basic.c:76\n+\t# left: 1\n+\t# right: 2\n+\tnot ok 3 - failing test\n+\tok 4 - failing test and assertion return 0\n+\tnot ok 5 - passing TEST_TODO() # TODO\n+\tok 6 - passing TEST_TODO() returns 1\n+\t# todo check ${SQ}check(x)${SQ} succeeded at t/unit-tests/t-basic.c:25\n+\tnot ok 7 - failing TEST_TODO()\n+\tok 8 - failing TEST_TODO() returns 0\n+\t# check \"0\" failed at t/unit-tests/t-basic.c:30\n+\t# skipping test - missing prerequisite\n+\t# skipping check ${SQ}1${SQ} at t/unit-tests/t-basic.c:32\n+\tok 9 - test_skip() # SKIP\n+\tok 10 - skipped test returns 1\n+\t# skipping test - missing prerequisite\n+\tok 11 - test_skip() inside TEST_TODO() # SKIP\n+\tok 12 - test_skip() inside TEST_TODO() returns 1\n+\t# check \"0\" failed at t/unit-tests/t-basic.c:48\n+\tnot ok 13 - TEST_TODO() after failing check\n+\tok 14 - TEST_TODO() after failing check returns 0\n+\t# check \"0\" failed at t/unit-tests/t-basic.c:56\n+\tnot ok 15 - failing check after TEST_TODO()\n+\tok 16 - failing check after TEST_TODO() returns 0\n+\t# check \"!strcmp(\"\\thello\\\\\\\\\", \"there\\\"\\n\")\" failed at t/unit-tests/t-basic.c:61\n+\t# left: \"\\011hello\\\\\\\\\"\n+\t# right: \"there\\\"\\012\"\n+\t# check \"!strcmp(\"NULL\", NULL)\" failed at t/unit-tests/t-basic.c:62\n+\t# left: \"NULL\"\n+\t# right: NULL\n+\t# check \"${SQ}a${SQ} == ${SQ}\\n${SQ}\" failed at t/unit-tests/t-basic.c:63\n+\t# left: ${SQ}a${SQ}\n+\t# right: ${SQ}\\012${SQ}\n+\t# check \"${SQ}\\\\\\\\${SQ} == ${SQ}\\\\${SQ}${SQ}\" failed at t/unit-tests/t-basic.c:64\n+\t# left: ${SQ}\\\\\\\\${SQ}\n+\t# right: ${SQ}\\\\${SQ}${SQ}\n+\tnot ok 17 - messages from failing string and char comparison\n+\t# BUG: test has no checks at t/unit-tests/t-basic.c:91\n+\tnot ok 18 - test with no checks\n+\tok 19 - test with no checks returns 0\n+\t1..19\n+\tEOF\n+\n+\t! \"$GIT_BUILD_DIR\"/t/unit-tests/bin/t-basic >actual &&\n+\ttest_cmp expect actual\n+'\n+\n+test_done\ndiff --git a/t/unit-tests/.gitignore b/t/unit-tests/.gitignore\nnew file mode 100644\nindex 0000000000..5e56e040ec\n--- /dev/null\n+++ b/t/unit-tests/.gitignore\n@@ -0,0 +1 @@\n+/bin\ndiff --git a/t/unit-tests/t-basic.c b/t/unit-tests/t-basic.c\nnew file mode 100644\nindex 0000000000..fda1ae59a6\n--- /dev/null\n+++ b/t/unit-tests/t-basic.c\n@@ -0,0 +1,95 @@\n+#include \"test-lib.h\"\n+\n+/*\n+ * The purpose of this \"unit test\" is to verify a few invariants of the unit\n+ * test framework itself, as well as to provide examples of output from actually\n+ * failing tests. As such, it is intended that this test fails, and thus it\n+ * should not be run as part of `make unit-tests`. Instead, we verify it behaves\n+ * as expected in the integration test t0080-unit-test-output.sh\n+ */\n+\n+/* Used to store the return value of check_int(). */\n+static int check_res;\n+\n+/* Used to store the return value of TEST(). */\n+static int test_res;\n+\n+static void t_res(int expect)\n+{\n+\tcheck_int(check_res, ==, expect);\n+\tcheck_int(test_res, ==, expect);\n+}\n+\n+static void t_todo(int x)\n+{\n+\tcheck_res = TEST_TODO(check(x));\n+}\n+\n+static void t_skip(void)\n+{\n+\tcheck(0);\n+\ttest_skip(\"missing prerequisite\");\n+\tcheck(1);\n+}\n+\n+static int do_skip(void)\n+{\n+\ttest_skip(\"missing prerequisite\");\n+\treturn 1;\n+}\n+\n+static void t_skip_todo(void)\n+{\n+\tcheck_res = TEST_TODO(do_skip());\n+}\n+\n+static void t_todo_after_fail(void)\n+{\n+\tcheck(0);\n+\tTEST_TODO(check(0));\n+}\n+\n+static void t_fail_after_todo(void)\n+{\n+\tcheck(1);\n+\tTEST_TODO(check(0));\n+\tcheck(0);\n+}\n+\n+static void t_messages(void)\n+{\n+\tcheck_str(\"\\thello\\\\\", \"there\\\"\\n\");\n+\tcheck_str(\"NULL\", NULL);\n+\tcheck_char('a', ==, '\\n');\n+\tcheck_char('\\\\', ==, '\\'');\n+}\n+\n+static void t_empty(void)\n+{\n+\t; /* empty */\n+}\n+\n+int cmd_main(int argc, const char **argv)\n+{\n+\ttest_res = TEST(check_res = check_int(1, ==, 1), \"passing test\");\n+\tTEST(t_res(1), \"passing test and assertion return 1\");\n+\ttest_res = TEST(check_res = check_int(1, ==, 2), \"failing test\");\n+\tTEST(t_res(0), \"failing test and assertion return 0\");\n+\ttest_res = TEST(t_todo(0), \"passing TEST_TODO()\");\n+\tTEST(t_res(1), \"passing TEST_TODO() returns 1\");\n+\ttest_res = TEST(t_todo(1), \"failing TEST_TODO()\");\n+\tTEST(t_res(0), \"failing TEST_TODO() returns 0\");\n+\ttest_res = TEST(t_skip(), \"test_skip()\");\n+\tTEST(check_int(test_res, ==, 1), \"skipped test returns 1\");\n+\ttest_res = TEST(t_skip_todo(), \"test_skip() inside TEST_TODO()\");\n+\tTEST(t_res(1), \"test_skip() inside TEST_TODO() returns 1\");\n+\ttest_res = TEST(t_todo_after_fail(), \"TEST_TODO() after failing check\");\n+\tTEST(check_int(test_res, ==, 0), \"TEST_TODO() after failing check returns 0\");\n+\ttest_res = TEST(t_fail_after_todo(), \"failing check after TEST_TODO()\");\n+\tTEST(check_int(test_res, ==, 0), \"failing check after TEST_TODO() returns 0\");\n+\tTEST(t_messages(), \"messages from failing string and char comparison\");\n+\ttest_res = TEST(t_empty(), \"test with no checks\");\n+\tTEST(check_int(test_res, ==, 0), \"test with no checks returns 0\");\n+\n+\treturn test_done();\n+}\ndiff --git a/t/unit-tests/t-strbuf.c b/t/unit-tests/t-strbuf.c\nnew file mode 100644\nindex 0000000000..de434a4441\n--- /dev/null\n+++ b/t/unit-tests/t-strbuf.c\n@@ -0,0 +1,120 @@\n+#include \"test-lib.h\"\n+#include \"strbuf.h\"\n+\n+/* wrapper that supplies tests with an empty, initialized strbuf */\n+static void setup(void (*f)(struct strbuf*, void*), void *data)\n+{\n+\tstruct strbuf buf = STRBUF_INIT;\n+\n+\tf(&buf, data);\n+\tstrbuf_release(&buf);\n+\tcheck_uint(buf.len, ==, 0);\n+\tcheck_uint(buf.alloc, ==, 0);\n+}\n+\n+/* wrapper that supplies tests with a populated, initialized strbuf */\n+static void setup_populated(void (*f)(struct strbuf*, void*), char *init_str, void *data)\n+{\n+\tstruct strbuf buf = STRBUF_INIT;\n+\n+\tstrbuf_addstr(&buf, init_str);\n+\tcheck_uint(buf.len, ==, strlen(init_str));\n+\tf(&buf, data);\n+\tstrbuf_release(&buf);\n+\tcheck_uint(buf.len, ==, 0);\n+\tcheck_uint(buf.alloc, ==, 0);\n+}\n+\n+static int assert_sane_strbuf(struct strbuf *buf)\n+{\n+\t/* Initialized strbufs should always have a non-NULL buffer */\n+\tif (!check(!!buf->buf))\n+\t\treturn 0;\n+\t/* Buffers should always be NUL-terminated */\n+\tif (!check_char(buf->buf[buf->len], ==, '\\0'))\n+\t\treturn 0;\n+\t/*\n+\t * Freshly-initialized strbufs may not have a dynamically allocated\n+\t * buffer\n+\t */\n+\tif (buf->len == 0 && buf->alloc == 0)\n+\t\treturn 1;\n+\t/* alloc must be at least one byte larger than len */\n+\treturn check_uint(buf->len, <, buf->alloc);\n+}\n+\n+static void t_static_init(void)\n+{\n+\tstruct strbuf buf = STRBUF_INIT;\n+\n+\tcheck_uint(buf.len, ==, 0);\n+\tcheck_uint(buf.alloc, ==, 0);\n+\tcheck_char(buf.buf[0], ==, '\\0');\n+}\n+\n+static void t_dynamic_init(void)\n+{\n+\tstruct strbuf buf;\n+\n+\tstrbuf_init(&buf, 1024);\n+\tcheck(assert_sane_strbuf(&buf));\n+\tcheck_uint(buf.len, ==, 0);\n+\tcheck_uint(buf.alloc, >=, 1024);\n+\tcheck_char(buf.buf[0], ==, '\\0');\n+\tstrbuf_release(&buf);\n+}\n+\n+static void t_addch(struct strbuf *buf, void *data)\n+{\n+\tconst char *p_ch = data;\n+\tconst char ch = *p_ch;\n+\tsize_t orig_alloc = buf->alloc;\n+\tsize_t orig_len = buf->len;\n+\n+\tif (!check(assert_sane_strbuf(buf)))\n+\t\treturn;\n+\tstrbuf_addch(buf, ch);\n+\tif (!check(assert_sane_strbuf(buf)))\n+\t\treturn;\n+\tif (!(check_uint(buf->len, ==, orig_len + 1) &&\n+\t check_uint(buf->alloc, >=, orig_alloc)))\n+\t\treturn; /* avoid de-referencing buf->buf */\n+\tcheck_char(buf->buf[buf->len - 1], ==, ch);\n+\tcheck_char(buf->buf[buf->len], ==, '\\0');\n+}\n+\n+static void t_addstr(struct strbuf *buf, void *data)\n+{\n+\tconst char *text = data;\n+\tsize_t len = strlen(text);\n+\tsize_t orig_alloc = buf->alloc;\n+\tsize_t orig_len = buf->len;\n+\n+\tif (!check(assert_sane_strbuf(buf)))\n+\t\treturn;\n+\tstrbuf_addstr(buf, text);\n+\tif (!check(assert_sane_strbuf(buf)))\n+\t\treturn;\n+\tif (!(check_uint(buf->len, ==, orig_len + len) &&\n+\t check_uint(buf->alloc, >=, orig_alloc) &&\n+\t check_uint(buf->alloc, >, orig_len + len) &&\n+\t check_char(buf->buf[orig_len + len], ==, '\\0')))\n+\t return;\n+\tcheck_str(buf->buf + orig_len, text);\n+}\n+\n+int cmd_main(int argc, const char **argv)\n+{\n+\tif (!TEST(t_static_init(), \"static initialization works\"))\n+\t\ttest_skip_all(\"STRBUF_INIT is broken\");\n+\tTEST(t_dynamic_init(), \"dynamic initialization works\");\n+\tTEST(setup(t_addch, \"a\"), \"strbuf_addch adds char\");\n+\tTEST(setup(t_addch, \"\"), \"strbuf_addch adds NUL char\");\n+\tTEST(setup_populated(t_addch, \"initial value\", \"a\"),\n+\t \"strbuf_addch appends to initial value\");\n+\tTEST(setup(t_addstr, \"hello there\"), \"strbuf_addstr adds string\");\n+\tTEST(setup_populated(t_addstr, \"initial value\", \"hello there\"),\n+\t \"strbuf_addstr appends string to initial value\");\n+\n+\treturn test_done();\n+}\ndiff --git a/t/unit-tests/test-lib.c b/t/unit-tests/test-lib.c\nnew file mode 100644\nindex 0000000000..b20f543121\n--- /dev/null\n+++ b/t/unit-tests/test-lib.c\n@@ -0,0 +1,329 @@\n+#include \"test-lib.h\"\n+\n+enum result {\n+\tRESULT_NONE,\n+\tRESULT_FAILURE,\n+\tRESULT_SKIP,\n+\tRESULT_SUCCESS,\n+\tRESULT_TODO\n+};\n+\n+static struct {\n+\tenum result result;\n+\tint count;\n+\tunsigned failed :1;\n+\tunsigned lazy_plan :1;\n+\tunsigned running :1;\n+\tunsigned skip_all :1;\n+\tunsigned todo :1;\n+} ctx = {\n+\t.lazy_plan = 1,\n+\t.result = RESULT_NONE,\n+};\n+\n+static void msg_with_prefix(const char *prefix, const char *format, va_list ap)\n+{\n+\tfflush(stderr);\n+\tif (prefix)\n+\t\tfprintf(stdout, \"%s\", prefix);\n+\tvprintf(format, ap); /* TODO: handle newlines */\n+\tputc('\\n', stdout);\n+\tfflush(stdout);\n+}\n+\n+void test_msg(const char *format, ...)\n+{\n+\tva_list ap;\n+\n+\tva_start(ap, format);\n+\tmsg_with_prefix(\"# \", format, ap);\n+\tva_end(ap);\n+}\n+\n+void test_plan(int count)\n+{\n+\tassert(!ctx.running);\n+\n+\tfflush(stderr);\n+\tprintf(\"1..%d\\n\", count);\n+\tfflush(stdout);\n+\tctx.lazy_plan = 0;\n+}\n+\n+int test_done(void)\n+{\n+\tassert(!ctx.running);\n+\n+\tif (ctx.lazy_plan)\n+\t\ttest_plan(ctx.count);\n+\n+\treturn ctx.failed;\n+}\n+\n+void test_skip(const char *format, ...)\n+{\n+\tva_list ap;\n+\n+\tassert(ctx.running);\n+\n+\tctx.result = RESULT_SKIP;\n+\tva_start(ap, format);\n+\tif (format)\n+\t\tmsg_with_prefix(\"# skipping test - \", format, ap);\n+\tva_end(ap);\n+}\n+\n+void test_skip_all(const char *format, ...)\n+{\n+\tva_list ap;\n+\tconst char *prefix;\n+\n+\tif (!ctx.count && ctx.lazy_plan) {\n+\t\t/* We have not printed a test plan yet */\n+\t\tprefix = \"1..0 # SKIP \";\n+\t\tctx.lazy_plan = 0;\n+\t} else {\n+\t\t/* We have already printed a test plan */\n+\t\tprefix = \"Bail out! # \";\n+\t\tctx.failed = 1;\n+\t}\n+\tctx.skip_all = 1;\n+\tctx.result = RESULT_SKIP;\n+\tva_start(ap, format);\n+\tmsg_with_prefix(prefix, format, ap);\n+\tva_end(ap);\n+}\n+\n+int test__run_begin(void)\n+{\n+\tassert(!ctx.running);\n+\n+\tctx.count++;\n+\tctx.result = RESULT_NONE;\n+\tctx.running = 1;\n+\n+\treturn ctx.skip_all;\n+}\n+\n+static void print_description(const char *format, va_list ap)\n+{\n+\tif (format) {\n+\t\tfputs(\" - \", stdout);\n+\t\tvprintf(format, ap);\n+\t}\n+}\n+\n+int test__run_end(int was_run UNUSED, const char *location, const char *format, ...)\n+{\n+\tva_list ap;\n+\n+\tassert(ctx.running);\n+\tassert(!ctx.todo);\n+\n+\tfflush(stderr);\n+\tva_start(ap, format);\n+\tif (!ctx.skip_all) {\n+\t\tswitch (ctx.result) {\n+\t\tcase RESULT_SUCCESS:\n+\t\t\tprintf(\"ok %d\", ctx.count);\n+\t\t\tprint_description(format, ap);\n+\t\t\tbreak;\n+\n+\t\tcase RESULT_FAILURE:\n+\t\t\tprintf(\"not ok %d\", ctx.count);\n+\t\t\tprint_description(format, ap);\n+\t\t\tbreak;\n+\n+\t\tcase RESULT_TODO:\n+\t\t\tprintf(\"not ok %d\", ctx.count);\n+\t\t\tprint_description(format, ap);\n+\t\t\tprintf(\" # TODO\");\n+\t\t\tbreak;\n+\n+\t\tcase RESULT_SKIP:\n+\t\t\tprintf(\"ok %d\", ctx.count);\n+\t\t\tprint_description(format, ap);\n+\t\t\tprintf(\" # SKIP\");\n+\t\t\tbreak;\n+\n+\t\tcase RESULT_NONE:\n+\t\t\ttest_msg(\"BUG: test has no checks at %s\", location);\n+\t\t\tprintf(\"not ok %d\", ctx.count);\n+\t\t\tprint_description(format, ap);\n+\t\t\tctx.result = RESULT_FAILURE;\n+\t\t\tbreak;\n+\t\t}\n+\t}\n+\tva_end(ap);\n+\tctx.running = 0;\n+\tif (ctx.skip_all)\n+\t\treturn 1;\n+\tputc('\\n', stdout);\n+\tfflush(stdout);\n+\tctx.failed |= ctx.result == RESULT_FAILURE;\n+\n+\treturn ctx.result != RESULT_FAILURE;\n+}\n+\n+static void test_fail(void)\n+{\n+\tassert(ctx.result != RESULT_SKIP);\n+\n+\tctx.result = RESULT_FAILURE;\n+}\n+\n+static void test_pass(void)\n+{\n+\tassert(ctx.result != RESULT_SKIP);\n+\n+\tif (ctx.result == RESULT_NONE)\n+\t\tctx.result = RESULT_SUCCESS;\n+}\n+\n+static void test_todo(void)\n+{\n+\tassert(ctx.result != RESULT_SKIP);\n+\n+\tif (ctx.result != RESULT_FAILURE)\n+\t\tctx.result = RESULT_TODO;\n+}\n+\n+int test_assert(const char *location, const char *check, int ok)\n+{\n+\tassert(ctx.running);\n+\n+\tif (ctx.result == RESULT_SKIP) {\n+\t\ttest_msg(\"skipping check '%s' at %s\", check, location);\n+\t\treturn 1;\n+\t} else if (!ctx.todo) {\n+\t\tif (ok) {\n+\t\t\ttest_pass();\n+\t\t} else {\n+\t\t\ttest_msg(\"check \\\"%s\\\" failed at %s\", check, location);\n+\t\t\ttest_fail();\n+\t\t}\n+\t}\n+\n+\treturn !!ok;\n+}\n+\n+void test__todo_begin(void)\n+{\n+\tassert(ctx.running);\n+\tassert(!ctx.todo);\n+\n+\tctx.todo = 1;\n+}\n+\n+int test__todo_end(const char *location, const char *check, int res)\n+{\n+\tassert(ctx.running);\n+\tassert(ctx.todo);\n+\n+\tctx.todo = 0;\n+\tif (ctx.result == RESULT_SKIP)\n+\t\treturn 1;\n+\tif (res) {\n+\t\ttest_msg(\"todo check '%s' succeeded at %s\", check, location);\n+\t\ttest_fail();\n+\t} else {\n+\t\ttest_todo();\n+\t}\n+\n+\treturn !res;\n+}\n+\n+int check_bool_loc(const char *loc, const char *check, int ok)\n+{\n+\treturn test_assert(loc, check, ok);\n+}\n+\n+union test__tmp test__tmp[2];\n+\n+int check_int_loc(const char *loc, const char *check, int ok,\n+\t\t intmax_t a, intmax_t b)\n+{\n+\tint ret = test_assert(loc, check, ok);\n+\n+\tif (!ret) {\n+\t\ttest_msg(\" left: %\"PRIdMAX, a);\n+\t\ttest_msg(\" right: %\"PRIdMAX, b);\n+\t}\n+\n+\treturn ret;\n+}\n+\n+int check_uint_loc(const char *loc, const char *check, int ok,\n+\t\t uintmax_t a, uintmax_t b)\n+{\n+\tint ret = test_assert(loc, check, ok);\n+\n+\tif (!ret) {\n+\t\ttest_msg(\" left: %\"PRIuMAX, a);\n+\t\ttest_msg(\" right: %\"PRIuMAX, b);\n+\t}\n+\n+\treturn ret;\n+}\n+\n+static void print_one_char(char ch, char quote)\n+{\n+\tif ((unsigned char)ch < 0x20u || ch == 0x7f) {\n+\t\t/* TODO: improve handling of \\a, \\b, \\f ... */\n+\t\tprintf(\"\\\\%03o\", (unsigned char)ch);\n+\t} else {\n+\t\tif (ch == '\\\\' || ch == quote)\n+\t\t\tputc('\\\\', stdout);\n+\t\tputc(ch, stdout);\n+\t}\n+}\n+\n+static void print_char(const char *prefix, char ch)\n+{\n+\tprintf(\"# %s: '\", prefix);\n+\tprint_one_char(ch, '\\'');\n+\tfputs(\"'\\n\", stdout);\n+}\n+\n+int check_char_loc(const char *loc, const char *check, int ok, char a, char b)\n+{\n+\tint ret = test_assert(loc, check, ok);\n+\n+\tif (!ret) {\n+\t\tfflush(stderr);\n+\t\tprint_char(\" left\", a);\n+\t\tprint_char(\" right\", b);\n+\t\tfflush(stdout);\n+\t}\n+\n+\treturn ret;\n+}\n+\n+static void print_str(const char *prefix, const char *str)\n+{\n+\tprintf(\"# %s: \", prefix);\n+\tif (!str) {\n+\t\tfputs(\"NULL\\n\", stdout);\n+\t} else {\n+\t\tputc('\"', stdout);\n+\t\twhile (*str)\n+\t\t\tprint_one_char(*str++, '\"');\n+\t\tfputs(\"\\\"\\n\", stdout);\n+\t}\n+}\n+\n+int check_str_loc(const char *loc, const char *check,\n+\t\t const char *a, const char *b)\n+{\n+\tint ok = (!a && !b) || (a && b && !strcmp(a, b));\n+\tint ret = test_assert(loc, check, ok);\n+\n+\tif (!ret) {\n+\t\tfflush(stderr);\n+\t\tprint_str(\" left\", a);\n+\t\tprint_str(\" right\", b);\n+\t\tfflush(stdout);\n+\t}\n+\n+\treturn ret;\n+}\ndiff --git a/t/unit-tests/test-lib.h b/t/unit-tests/test-lib.h\nnew file mode 100644\nindex 0000000000..a8f07ae0b7\n--- /dev/null\n+++ b/t/unit-tests/test-lib.h\n@@ -0,0 +1,149 @@\n+#ifndef TEST_LIB_H\n+#define TEST_LIB_H\n+\n+#include \"git-compat-util.h\"\n+\n+/*\n+ * Run a test function, returns 1 if the test succeeds, 0 if it\n+ * fails. If test_skip_all() has been called then the test will not be\n+ * run. The description for each test should be unique. For example:\n+ *\n+ * TEST(test_something(arg1, arg2), \"something %d %d\", arg1, arg2)\n+ */\n+#define TEST(t, ...)\t\t\t\t\t\\\n+\ttest__run_end(test__run_begin() ? 0 : (t, 1),\t\\\n+\t\t TEST_LOCATION(), __VA_ARGS__)\n+\n+/*\n+ * Print a test plan, should be called before any tests. If the number\n+ * of tests is not known in advance test_done() will automatically\n+ * print a plan at the end of the test program.\n+ */\n+void test_plan(int count);\n+\n+/*\n+ * test_done() must be called at the end of main(). It will print the\n+ * plan if plan() was not called at the beginning of the test program\n+ * and returns the exit code for the test program.\n+ */\n+int test_done(void);\n+\n+/* Skip the current test. */\n+__attribute__((format (printf, 1, 2)))\n+void test_skip(const char *format, ...);\n+\n+/* Skip all remaining tests. */\n+__attribute__((format (printf, 1, 2)))\n+void test_skip_all(const char *format, ...);\n+\n+/* Print a diagnostic message to stdout. */\n+__attribute__((format (printf, 1, 2)))\n+void test_msg(const char *format, ...);\n+\n+/*\n+ * Test checks are built around test_assert(). checks return 1 on\n+ * success, 0 on failure. If any check fails then the test will fail. To\n+ * create a custom check define a function that wraps test_assert() and\n+ * a macro to wrap that function to provide a source location and\n+ * stringified arguments. Custom checks that take pointer arguments\n+ * should be careful to check that they are non-NULL before\n+ * dereferencing them. For example:\n+ *\n+ * static int check_oid_loc(const char *loc, const char *check,\n+ *\t\t\t struct object_id *a, struct object_id *b)\n+ * {\n+ *\t int res = test_assert(loc, check, a && b && oideq(a, b));\n+ *\n+ *\t if (!res) {\n+ *\t\t test_msg(\" left: %s\", a ? oid_to_hex(a) : \"NULL\";\n+ *\t\t test_msg(\" right: %s\", b ? oid_to_hex(a) : \"NULL\";\n+ *\n+ *\t }\n+ *\t return res;\n+ * }\n+ *\n+ * #define check_oid(a, b) \\\n+ *\t check_oid_loc(TEST_LOCATION(), \"oideq(\"#a\", \"#b\")\", a, b)\n+ */\n+int test_assert(const char *location, const char *check, int ok);\n+\n+/* Helper macro to pass the location to checks */\n+#define TEST_LOCATION() TEST__MAKE_LOCATION(__LINE__)\n+\n+/* Check a boolean condition. */\n+#define check(x)\t\t\t\t\\\n+\tcheck_bool_loc(TEST_LOCATION(), #x, x)\n+int check_bool_loc(const char *loc, const char *check, int ok);\n+\n+/*\n+ * Compare two integers. Prints a message with the two values if the\n+ * comparison fails. NB this is not thread safe.\n+ */\n+#define check_int(a, op, b)\t\t\t\t\t\t\\\n+\t(test__tmp[0].i = (a), test__tmp[1].i = (b),\t\t\t\\\n+\t check_int_loc(TEST_LOCATION(), #a\" \"#op\" \"#b,\t\t\t\\\n+\t\t test__tmp[0].i op test__tmp[1].i,\t\t\\\n+\t\t test__tmp[0].i, test__tmp[1].i))\n+int check_int_loc(const char *loc, const char *check, int ok,\n+\t\t intmax_t a, intmax_t b);\n+\n+/*\n+ * Compare two unsigned integers. Prints a message with the two values\n+ * if the comparison fails. NB this is not thread safe.\n+ */\n+#define check_uint(a, op, b)\t\t\t\t\t\t\\\n+\t(test__tmp[0].u = (a), test__tmp[1].u = (b),\t\t\t\\\n+\t check_uint_loc(TEST_LOCATION(), #a\" \"#op\" \"#b,\t\t\t\\\n+\t\t\ttest__tmp[0].u op test__tmp[1].u,\t\t\\\n+\t\t\ttest__tmp[0].u, test__tmp[1].u))\n+int check_uint_loc(const char *loc, const char *check, int ok,\n+\t\t uintmax_t a, uintmax_t b);\n+\n+/*\n+ * Compare two chars. Prints a message with the two values if the\n+ * comparison fails. NB this is not thread safe.\n+ */\n+#define check_char(a, op, b)\t\t\t\t\t\t\\\n+\t(test__tmp[0].c = (a), test__tmp[1].c = (b),\t\t\t\\\n+\t check_char_loc(TEST_LOCATION(), #a\" \"#op\" \"#b,\t\t\t\\\n+\t\t\ttest__tmp[0].c op test__tmp[1].c,\t\t\\\n+\t\t\ttest__tmp[0].c, test__tmp[1].c))\n+int check_char_loc(const char *loc, const char *check, int ok,\n+\t\t char a, char b);\n+\n+/* Check whether two strings are equal. */\n+#define check_str(a, b)\t\t\t\t\t\t\t\\\n+\tcheck_str_loc(TEST_LOCATION(), \"!strcmp(\"#a\", \"#b\")\", a, b)\n+int check_str_loc(const char *loc, const char *check,\n+\t\t const char *a, const char *b);\n+\n+/*\n+ * Wrap a check that is known to fail. If the check succeeds then the\n+ * test will fail. Returns 1 if the check fails, 0 if it\n+ * succeeds. For example:\n+ *\n+ * TEST_TODO(check(0));\n+ */\n+#define TEST_TODO(check) \\\n+\t(test__todo_begin(), test__todo_end(TEST_LOCATION(), #check, check))\n+\n+/* Private helpers */\n+\n+#define TEST__STR(x) #x\n+#define TEST__MAKE_LOCATION(line) __FILE__ \":\" TEST__STR(line)\n+\n+union test__tmp {\n+\tintmax_t i;\n+\tuintmax_t u;\n+\tchar c;\n+};\n+\n+extern union test__tmp test__tmp[2];\n+\n+int test__run_begin(void);\n+__attribute__((format (printf, 3, 4)))\n+int test__run_end(int, const char *, const char *, ...);\n+void test__todo_begin(void);\n+int test__todo_end(const char *, const char *, int);\n+\n+#endif /* TEST_LIB_H */\n-- \n2.42.0.869.gea05f2083d-goog\n\n"},{"id":"484324","messageId":"xmqqmsvxox08.fsf@gitster.g","threadId":"59937","inReplyTo":"ZUKQ8xtFSYbSBPXb@google.com","subject":"Re: [PATCH v8 2.5/3] fixup! unit tests: add TAP unit test framework","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2023-11-01T23:48:55Z","receivedAt":"2023-11-01T23:49:02Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Josh Steadmon writes:\n\n> On 2023.10.16 09:41, Junio C Hamano wrote:\n>> Phillip Wood writes:\n>> \n>> > From: Phillip Wood \n>> >\n>> > Here are a couple of cleanups for the unit test framework that I\n>> > noticed.\n>> \n>> Thanks. I trust that this will be squashed into the next update,\n>> but in the meantime, I'll include it in the copy of the series I\n>> have (without squashing). Here is another one I noticed.\n>> \n>> ----- >8 --------- >8 --------- >8 -----\n>> Subject: [PATCH] fixup! ci: run unit tests in CI\n>> \n>> A CI job failed due to contrib/coccinelle/equals-null.cocci\n>> and suggested this change, which seems sensible.\n>> \n>> Signed-off-by: Junio C Hamano \n>> ---\n>> t/unit-tests/t-strbuf.c | 2 +-\n>> 1 file changed, 1 insertion(+), 1 deletion(-)\n>\n> Applied in v9, thanks!\n\nThanks for working well together.\n"},{"id":"484325","messageId":"xmqqil6lowzi.fsf@gitster.g","threadId":"59937","inReplyTo":"ZUKPJsW1mT62Mcjy@google.com","subject":"Re: [PATCH v8 1/3] unit tests: Add a project plan document","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2023-11-01T23:49:21Z","receivedAt":"2023-11-01T23:49:28Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Josh Steadmon writes:\n\n> On 2023.10.27 22:12, Christian Couder wrote:\n>> On Tue, Oct 10, 2023 at 12:22 AM Josh Steadmon wrote:\n>> >\n>> > In our current testing environment, we spend a significant amount of\n>> > effort crafting end-to-end tests for error conditions that could easily\n>> > be captured by unit tests (or we simply forgo some hard-to-setup and\n>> > rare error conditions). Describe what we hope to accomplish by\n>> > implementing unit tests, and explain some open questions and milestones.\n>> > Discuss desired features for test frameworks/harnesses, and provide a\n>> > preliminary comparison of several different frameworks.\n>> \n>> Nit: Not sure why the test framework comparison is \"preliminary\" as we\n>> have actually selected a unit test framework and are adding it in the\n>> next patch of the series. I understand that this was perhaps written\n>> before the choice was made, but maybe we might want to update that\n>> now.\n>\n> Fixed in v9, thanks.\n\nThanks for working well together.\n"},{"id":"484326","messageId":"xmqqedh9owz4.fsf@gitster.g","threadId":"59937","inReplyTo":"ZUKQ2eYmfMEhc_gH@google.com","subject":"Re: [PATCH v8 2.5/3] fixup! unit tests: add TAP unit test framework","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2023-11-01T23:49:35Z","receivedAt":"2023-11-01T23:49:46Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Josh Steadmon writes:\n\n> On 2023.10.16 14:43, Phillip Wood wrote:\n>> From: Phillip Wood \n>> \n>> Here are a couple of cleanups for the unit test framework that I\n>> noticed.\n>> \n>> Update the documentation of the example custom check to reflect the\n>> change in return value of test_assert() and mention that\n>> checks should be careful when dereferencing pointer arguments.\n>> \n>> Also avoid evaluating macro augments twice in check_int() and\n>> friends. The global variable test__tmp was introduced to avoid\n>> evaluating the arguments to these macros more than once but the macros\n>> failed to use it when passing the values being compared to\n>> check_int_loc().\n>> \n>> Signed-off-by: Phillip Wood \n>> ---\n>> t/unit-tests/test-lib.h | 26 ++++++++++++++++----------\n>> 1 file changed, 16 insertions(+), 10 deletions(-)\n>\n> Applied in v9, thanks!\n\nThanks for working well together.\n"},{"id":"484416","messageId":"CAP8UFD3SVnu+HFQhFpsF4PA6pK5B5L+aP-jxRX=Ro3EYekS0kg@mail.gmail.com","threadId":"59937","inReplyTo":"8b831f4937a167519b77c724fcbc36884bbfe43d.1698881249.git.steadmon@google.com","subject":"Re: [PATCH v9 2/3] unit tests: add TAP unit test framework","fromName":"Christian Couder","fromEmail":"christian.couder@gmail.com","sentAt":"2023-11-03T21:54:22Z","receivedAt":"2023-11-03T21:54:38Z","isPatch":true,"sender":{"key":"christian.couder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/208954?v=4"},"body":"On Thu, Nov 2, 2023 at 12:31 AM Josh Steadmon wrote:\n>\n> From: Phillip Wood \n\n> +int test_assert(const char *location, const char *check, int ok)\n> +{\n> + assert(ctx.running);\n> +\n> + if (ctx.result == RESULT_SKIP) {\n> + test_msg(\"skipping check '%s' at %s\", check, location);\n> + return 1;\n> + } else if (!ctx.todo) {\n\nI suggested removing the \"else\" and moving the \"if (!ctx.todo) {\" to\nits own line in the previous round and thought you agreed with that,\nbut maybe it fell through the cracks somehow.\n\nAnyway I think this is a minor nit, and the series looks good to me.\n\n> + if (ok) {\n> + test_pass();\n> + } else {\n> + test_msg(\"check \\\"%s\\\" failed at %s\", check, location);\n> + test_fail();\n> + }\n> + }\n> +\n> + return !!ok;\n> +}\n"},{"id":"484650","messageId":"ZU0cFK1lrBdXpHug@google.com","threadId":"59937","inReplyTo":"CAP8UFD3SVnu+HFQhFpsF4PA6pK5B5L+aP-jxRX=Ro3EYekS0kg@mail.gmail.com","subject":"Re: [PATCH v9 2/3] unit tests: add TAP unit test framework","fromName":"Josh Steadmon","fromEmail":"steadmon@google.com","sentAt":"2023-11-09T17:51:16Z","receivedAt":"2023-11-09T17:51:24Z","isPatch":true,"sender":{"key":"steadmon@google.com","avatar":"https://avatars.githubusercontent.com/u/2654920?v=4"},"body":"On 2023.11.03 22:54, Christian Couder wrote:\n> On Thu, Nov 2, 2023 at 12:31 AM Josh Steadmon wrote:\n> >\n> > From: Phillip Wood \n> \n> > +int test_assert(const char *location, const char *check, int ok)\n> > +{\n> > + assert(ctx.running);\n> > +\n> > + if (ctx.result == RESULT_SKIP) {\n> > + test_msg(\"skipping check '%s' at %s\", check, location);\n> > + return 1;\n> > + } else if (!ctx.todo) {\n> \n> I suggested removing the \"else\" and moving the \"if (!ctx.todo) {\" to\n> its own line in the previous round and thought you agreed with that,\n> but maybe it fell through the cracks somehow.\n> \n> Anyway I think this is a minor nit, and the series looks good to me.\n\nAhh, sorry about that, I must have accidentally dropped a fixup patch at\nsome point. I'll correct that and send v10 soon.\n"},{"id":"484655","messageId":"cover.1699555664.git.steadmon@google.com","threadId":"59937","inReplyTo":"0169ce6fb9ccafc089b74ae406db0d1a8ff8ac65.1688165272.git.steadmon@google.com","subject":"[PATCH v10 0/3] Add unit test framework and project plan","fromName":"Josh Steadmon","fromEmail":"steadmon@google.com","sentAt":"2023-11-09T18:50:41Z","receivedAt":"2023-11-09T18:50:49Z","isPatch":true,"sender":{"key":"steadmon@google.com","avatar":"https://avatars.githubusercontent.com/u/2654920?v=4"},"body":"In our current testing environment, we spend a significant amount of\neffort crafting end-to-end tests for error conditions that could easily\nbe captured by unit tests (or we simply forgo some hard-to-setup and\nrare error conditions). Unit tests additionally provide stability to the\ncodebase and can simplify debugging through isolation. Turning parts of\nGit into libraries[1] gives us the ability to run unit tests on the\nlibraries and to write unit tests in C. Writing unit tests in pure C,\nrather than with our current shell/test-tool helper setup, simplifies\ntest setup, simplifies passing data around (no shell-isms required), and\nreduces testing runtime by not spawning a separate process for every\ntest invocation.\n\nThis series begins with a project document covering our goals for adding\nunit tests and a discussion of alternative frameworks considered, as\nwell as the features used to evaluate them. A rendered preview of this\ndoc can be found at [2]. It also adds Phillip Wood's TAP implemenation\n(with some slightly re-worked Makefile rules) and a sample strbuf unit\ntest. Finally, we modify the configs for GitHub and Cirrus CI to run the\nunit tests. Sample runs showing successful CI runs can be found at [3],\n[4], and [5].\n\n[1] https://lore.kernel.org/git/CAJoAoZ=Cig_kLocxKGax31sU7Xe4==BGzC__Bg2_pr7krNq6MA@mail.gmail.com/\n[2] https://github.com/steadmon/git/blob/unit-tests-asciidoc/Documentation/technical/unit-tests.adoc\n[3] https://github.com/steadmon/git/actions/runs/5884659246/job/15959781385#step:4:1803\n[4] https://github.com/steadmon/git/actions/runs/5884659246/job/15959938401#step:5:186\n[5] https://cirrus-ci.com/task/6126304366428160 (unrelated tests failed,\n but note that t-strbuf ran successfully)\n\nChanges in v10:\n- Included a promised style cleanup in test-lib.c that was accidentally\n dropped in v9.\n\nChanges in v9:\n- Included some asciidoc cleanups suggested by Oswald Buddenhagen.\n- Applied a style fixup that Coccinelle complained about.\n- Applied some NULL-safety fixups.\n- Used check_*() more widely in t-strbuf helper functions\n\nChanges in v8:\n- Flipped return values for TEST, TEST_TODO, and check_* macros &\n functions. This makes it easier to reason about control flow for\n patterns like:\n if (check(some_condition)) { ... }\n- Moved unit test binaries to t/unit-tests/bin to simplify .gitignore\n patterns.\n- Removed testing of some strbuf implementation details in t-strbuf.c\n\nChanges in v7:\n- Fix corrupt diff in patch #2, sorry for the noise.\n\nChanges in v6:\n- Officially recommend using Phillip Wood's TAP framework\n- Add an example strbuf unit test using the TAP framework as well as\n Makefile integration\n- Run unit tests in CI\n\nChanges in v5:\n- Add comparison point \"License\".\n- Discuss feature priorities\n- Drop frameworks:\n - Incompatible licenses: libtap, cmocka\n - Missing source: MyTAP\n - No TAP support: µnit, cmockery, cmockery2, Unity, minunit, CUnit\n- Drop comparison point \"Coverage reports\": this can generally be\n handled by tools such as `gcov` regardless of the framework used.\n- Drop comparison point \"Inline tests\": there didn't seem to be\n strong interest from reviewers for this feature.\n- Drop comparison point \"Scheduling / re-running\": this was not\n supported by any of the main contenders, and is generally better\n handled by the harness rather than framework.\n- Drop comparison point \"Lazy test planning\": this was supported by\n all frameworks that provide TAP output.\n\nChanges in v4:\n- Add link anchors for the framework comparison dimensions\n- Explain \"Partial\" results for each dimension\n- Use consistent dimension names in the section headers and comparison\n tables\n- Add \"Project KLOC\", \"Adoption\", and \"Inline tests\" dimensions\n- Fill in a few of the missing entries in the comparison table\n\nChanges in v3:\n- Expand the doc with discussion of desired features and a WIP\n comparison.\n- Drop all implementation patches until a framework is selected.\n- Link to v2: https://lore.kernel.org/r/20230517-unit-tests-v2-v2-0-21b5b60f4b32@google.com\n\n\nJosh Steadmon (2):\n unit tests: Add a project plan document\n ci: run unit tests in CI\n\nPhillip Wood (1):\n unit tests: add TAP unit test framework\n\n .cirrus.yml | 2 +-\n Documentation/Makefile | 1 +\n Documentation/technical/unit-tests.txt | 240 ++++++++++++++++++\n Makefile | 28 ++-\n ci/run-build-and-tests.sh | 2 +\n ci/run-test-slice.sh | 5 +\n t/Makefile | 15 +-\n t/t0080-unit-test-output.sh | 58 +++++\n t/unit-tests/.gitignore | 1 +\n t/unit-tests/t-basic.c | 95 +++++++\n t/unit-tests/t-strbuf.c | 120 +++++++++\n t/unit-tests/test-lib.c | 330 +++++++++++++++++++++++++\n t/unit-tests/test-lib.h | 149 +++++++++++\n 13 files changed, 1041 insertions(+), 5 deletions(-)\n create mode 100644 Documentation/technical/unit-tests.txt\n create mode 100755 t/t0080-unit-test-output.sh\n create mode 100644 t/unit-tests/.gitignore\n create mode 100644 t/unit-tests/t-basic.c\n create mode 100644 t/unit-tests/t-strbuf.c\n create mode 100644 t/unit-tests/test-lib.c\n create mode 100644 t/unit-tests/test-lib.h\n\nRange-diff against v9:\n-: ---------- > 1: f706ba9b68 unit tests: Add a project plan document\n1: 8b831f4937 ! 2: 7a5e21bcff unit tests: add TAP unit test framework\n @@ t/unit-tests/test-lib.c (new)\n +\tif (ctx.result == RESULT_SKIP) {\n +\t\ttest_msg(\"skipping check '%s' at %s\", check, location);\n +\t\treturn 1;\n -+\t} else if (!ctx.todo) {\n ++\t}\n ++\tif (!ctx.todo) {\n +\t\tif (ok) {\n +\t\t\ttest_pass();\n +\t\t} else {\n2: 08d27bb5f9 = 3: 0129ec062c ci: run unit tests in CI\n\nbase-commit: a9e066fa63149291a55f383cfa113d8bdbdaa6b3\n-- \n2.42.0.869.gea05f2083d-goog\n\n"},{"id":"484656","messageId":"f706ba9b682e7c4070d49086ad3af582bf269c79.1699555664.git.steadmon@google.com","threadId":"59937","inReplyTo":"cover.1699555664.git.steadmon@google.com","subject":"[PATCH v10 1/3] unit tests: Add a project plan document","fromName":"Josh Steadmon","fromEmail":"steadmon@google.com","sentAt":"2023-11-09T18:50:42Z","receivedAt":"2023-11-09T18:50:51Z","isPatch":true,"sender":{"key":"steadmon@google.com","avatar":"https://avatars.githubusercontent.com/u/2654920?v=4"},"body":"In our current testing environment, we spend a significant amount of\neffort crafting end-to-end tests for error conditions that could easily\nbe captured by unit tests (or we simply forgo some hard-to-setup and\nrare error conditions). Describe what we hope to accomplish by\nimplementing unit tests, and explain some open questions and milestones.\nDiscuss desired features for test frameworks/harnesses, and provide a\ncomparison of several different frameworks. Finally, document our\nrationale for implementing a custom framework.\n\nCo-authored-by: Calvin Wan \nSigned-off-by: Calvin Wan \nSigned-off-by: Josh Steadmon \n---\n Documentation/Makefile | 1 +\n Documentation/technical/unit-tests.txt | 240 +++++++++++++++++++++++++\n 2 files changed, 241 insertions(+)\n create mode 100644 Documentation/technical/unit-tests.txt\n\ndiff --git a/Documentation/Makefile b/Documentation/Makefile\nindex b629176d7d..3f2383a12c 100644\n--- a/Documentation/Makefile\n+++ b/Documentation/Makefile\n@@ -122,6 +122,7 @@ TECH_DOCS += technical/scalar\n TECH_DOCS += technical/send-pack-pipeline\n TECH_DOCS += technical/shallow\n TECH_DOCS += technical/trivial-merge\n+TECH_DOCS += technical/unit-tests\n SP_ARTICLES += $(TECH_DOCS)\n SP_ARTICLES += technical/api-index\n \ndiff --git a/Documentation/technical/unit-tests.txt b/Documentation/technical/unit-tests.txt\nnew file mode 100644\nindex 0000000000..206037ffb1\n--- /dev/null\n+++ b/Documentation/technical/unit-tests.txt\n@@ -0,0 +1,240 @@\n+= Unit Testing\n+\n+In our current testing environment, we spend a significant amount of effort\n+crafting end-to-end tests for error conditions that could easily be captured by\n+unit tests (or we simply forgo some hard-to-setup and rare error conditions).\n+Unit tests additionally provide stability to the codebase and can simplify\n+debugging through isolation. Writing unit tests in pure C, rather than with our\n+current shell/test-tool helper setup, simplifies test setup, simplifies passing\n+data around (no shell-isms required), and reduces testing runtime by not\n+spawning a separate process for every test invocation.\n+\n+We believe that a large body of unit tests, living alongside the existing test\n+suite, will improve code quality for the Git project.\n+\n+== Definitions\n+\n+For the purposes of this document, we'll use *test framework* to refer to\n+projects that support writing test cases and running tests within the context\n+of a single executable. *Test harness* will refer to projects that manage\n+running multiple executables (each of which may contain multiple test cases) and\n+aggregating their results.\n+\n+In reality, these terms are not strictly defined, and many of the projects\n+discussed below contain features from both categories.\n+\n+For now, we will evaluate projects solely on their framework features. Since we\n+are relying on having TAP output (see below), we can assume that any framework\n+can be made to work with a harness that we can choose later.\n+\n+\n+== Summary\n+\n+We believe the best way forward is to implement a custom TAP framework for the\n+Git project. We use a version of the framework originally proposed in\n+https://lore.kernel.org/git/c902a166-98ce-afba-93f2-ea6027557176@gmail.com/[1].\n+\n+See the <> section below for the\n+rationale behind this decision.\n+\n+\n+== Choosing a test harness\n+\n+During upstream discussion, it was occasionally noted that `prove` provides many\n+convenient features, such as scheduling slower tests first, or re-running\n+previously failed tests.\n+\n+While we already support the use of `prove` as a test harness for the shell\n+tests, it is not strictly required. The t/Makefile allows running shell tests\n+directly (though with interleaved output if parallelism is enabled). Git\n+developers who wish to use `prove` as a more advanced harness can do so by\n+setting DEFAULT_TEST_TARGET=prove in their config.mak.\n+\n+We will follow a similar approach for unit tests: by default the test\n+executables will be run directly from the t/Makefile, but `prove` can be\n+configured with DEFAULT_UNIT_TEST_TARGET=prove.\n+\n+\n+[[framework-selection]]\n+== Framework selection\n+\n+There are a variety of features we can use to rank the candidate frameworks, and\n+those features have different priorities:\n+\n+* Critical features: we probably won't consider a framework without these\n+** Can we legally / easily use the project?\n+*** <>\n+*** <>\n+*** <>\n+*** <>\n+** Does the project support our bare-minimum needs?\n+*** <>\n+*** <>\n+*** <>\n+* Nice-to-have features:\n+** <>\n+** <>\n+** <>\n+* Tie-breaker stats\n+** <>\n+** <>\n+\n+[[license]]\n+=== License\n+\n+We must be able to legally use the framework in connection with Git. As Git is\n+licensed only under GPLv2, we must eliminate any LGPLv3, GPLv3, or Apache 2.0\n+projects.\n+\n+[[vendorable-or-ubiquitous]]\n+=== Vendorable or ubiquitous\n+\n+We want to avoid forcing Git developers to install new tools just to run unit\n+tests. Any prospective frameworks and harnesses must either be vendorable\n+(meaning, we can copy their source directly into Git's repository), or so\n+ubiquitous that it is reasonable to expect that most developers will have the\n+tools installed already.\n+\n+[[maintainable-extensible]]\n+=== Maintainable / extensible\n+\n+It is unlikely that any pre-existing project perfectly fits our needs, so any\n+project we select will need to be actively maintained and open to accepting\n+changes. Alternatively, assuming we are vendoring the source into our repo, it\n+must be simple enough that Git developers can feel comfortable making changes as\n+needed to our version.\n+\n+In the comparison table below, \"True\" means that the framework seems to have\n+active developers, that it is simple enough that Git developers can make changes\n+to it, and that the project seems open to accepting external contributions (or\n+that it is vendorable). \"Partial\" means that at least one of the above\n+conditions holds.\n+\n+[[major-platform-support]]\n+=== Major platform support\n+\n+At a bare minimum, unit-testing must work on Linux, MacOS, and Windows.\n+\n+In the comparison table below, \"True\" means that it works on all three major\n+platforms with no issues. \"Partial\" means that there may be annoyances on one or\n+more platforms, but it is still usable in principle.\n+\n+[[tap-support]]\n+=== TAP support\n+\n+The https://testanything.org/[Test Anything Protocol] is a text-based interface\n+that allows tests to communicate with a test harness. It is already used by\n+Git's integration test suite. Supporting TAP output is a mandatory feature for\n+any prospective test framework.\n+\n+In the comparison table below, \"True\" means this is natively supported.\n+\"Partial\" means TAP output must be generated by post-processing the native\n+output.\n+\n+Frameworks that do not have at least Partial support will not be evaluated\n+further.\n+\n+[[diagnostic-output]]\n+=== Diagnostic output\n+\n+When a test case fails, the framework must generate enough diagnostic output to\n+help developers find the appropriate test case in source code in order to debug\n+the failure.\n+\n+[[runtime-skippable-tests]]\n+=== Runtime-skippable tests\n+\n+Test authors may wish to skip certain test cases based on runtime circumstances,\n+so the framework should support this.\n+\n+[[parallel-execution]]\n+=== Parallel execution\n+\n+Ideally, we will build up a significant collection of unit test cases, most\n+likely split across multiple executables. It will be necessary to run these\n+tests in parallel to enable fast develop-test-debug cycles.\n+\n+In the comparison table below, \"True\" means that individual test cases within a\n+single test executable can be run in parallel. We assume that executable-level\n+parallelism can be handled by the test harness.\n+\n+[[mock-support]]\n+=== Mock support\n+\n+Unit test authors may wish to test code that interacts with objects that may be\n+inconvenient to handle in a test (e.g. interacting with a network service).\n+Mocking allows test authors to provide a fake implementation of these objects\n+for more convenient tests.\n+\n+[[signal-error-handling]]\n+=== Signal & error handling\n+\n+The test framework should fail gracefully when test cases are themselves buggy\n+or when they are interrupted by signals during runtime.\n+\n+[[project-kloc]]\n+=== Project KLOC\n+\n+The size of the project, in thousands of lines of code as measured by\n+https://dwheeler.com/sloccount/[sloccount] (rounded up to the next multiple of\n+1,000). As a tie-breaker, we probably prefer a project with fewer LOC.\n+\n+[[adoption]]\n+=== Adoption\n+\n+As a tie-breaker, we prefer a more widely-used project. We use the number of\n+GitHub / GitLab stars to estimate this.\n+\n+\n+=== Comparison\n+\n+:true: [lime-background]#True#\n+:false: [red-background]#False#\n+:partial: [yellow-background]#Partial#\n+\n+:gpl: [lime-background]#GPL v2#\n+:isc: [lime-background]#ISC#\n+:mit: [lime-background]#MIT#\n+:expat: [lime-background]#Expat#\n+:lgpl: [lime-background]#LGPL v2.1#\n+\n+:custom-impl: https://lore.kernel.org/git/c902a166-98ce-afba-93f2-ea6027557176@gmail.com/[Custom Git impl.]\n+:greatest: https://github.com/silentbicycle/greatest[Greatest]\n+:criterion: https://github.com/Snaipe/Criterion[Criterion]\n+:c-tap: https://github.com/rra/c-tap-harness/[C TAP]\n+:check: https://libcheck.github.io/check/[Check]\n+\n+[format=\"csv\",options=\"header\",width=\"33%\",subs=\"specialcharacters,attributes,quotes,macros\"]\n+|=====\n+Framework,\"<>\",\"<>\",\"<>\",\"<>\",\"<>\",\"<>\",\"<>\",\"<>\",\"<>\",\"<>\",\"<>\",\"<>\"\n+{custom-impl},{gpl},{true},{true},{true},{true},{true},{true},{false},{false},{false},1,0\n+{greatest},{isc},{true},{partial},{true},{partial},{true},{true},{false},{false},{false},3,1400\n+{criterion},{mit},{false},{partial},{true},{true},{true},{true},{true},{false},{true},19,1800\n+{c-tap},{expat},{true},{partial},{partial},{true},{false},{true},{false},{false},{false},4,33\n+{check},{lgpl},{false},{partial},{true},{true},{true},{false},{false},{false},{true},17,973\n+|=====\n+\n+=== Additional framework candidates\n+\n+Several suggested frameworks have been eliminated from consideration:\n+\n+* Incompatible licenses:\n+** https://github.com/zorgnax/libtap[libtap] (LGPL v3)\n+** https://cmocka.org/[cmocka] (Apache 2.0)\n+* Missing source: https://www.kindahl.net/mytap/doc/index.html[MyTap]\n+* No TAP support:\n+** https://nemequ.github.io/munit/[µnit]\n+** https://github.com/google/cmockery[cmockery]\n+** https://github.com/lpabon/cmockery2[cmockery2]\n+** https://github.com/ThrowTheSwitch/Unity[Unity]\n+** https://github.com/siu/minunit[minunit]\n+** https://cunit.sourceforge.net/[CUnit]\n+\n+\n+== Milestones\n+\n+* Add useful tests of library-like code\n+* Integrate with\n+ https://lore.kernel.org/git/20230502211454.1673000-1-calvinwan@google.com/[stdlib\n+ work]\n+* Run alongside regular `make test` target\n-- \n2.42.0.869.gea05f2083d-goog\n\n"},{"id":"484657","messageId":"7a5e21bcff577b9ffaa01ffd8e96d07a0a293ac0.1699555664.git.steadmon@google.com","threadId":"59937","inReplyTo":"cover.1699555664.git.steadmon@google.com","subject":"[PATCH v10 2/3] unit tests: add TAP unit test framework","fromName":"Josh Steadmon","fromEmail":"steadmon@google.com","sentAt":"2023-11-09T18:50:43Z","receivedAt":"2023-11-09T18:50:53Z","isPatch":true,"sender":{"key":"steadmon@google.com","avatar":"https://avatars.githubusercontent.com/u/2654920?v=4"},"body":"From: Phillip Wood \n\nThis patch contains an implementation for writing unit tests with TAP\noutput. Each test is a function that contains one or more checks. The\ntest is run with the TEST() macro and if any of the checks fail then the\ntest will fail. A complete program that tests STRBUF_INIT would look\nlike\n\n #include \"test-lib.h\"\n #include \"strbuf.h\"\n\n static void t_static_init(void)\n {\n struct strbuf buf = STRBUF_INIT;\n\n check_uint(buf.len, ==, 0);\n check_uint(buf.alloc, ==, 0);\n check_char(buf.buf[0], ==, '\\0');\n }\n\n int main(void)\n {\n TEST(t_static_init(), \"static initialization works);\n\n return test_done();\n }\n\nThe output of this program would be\n\n ok 1 - static initialization works\n 1..1\n\nIf any of the checks in a test fail then they print a diagnostic message\nto aid debugging and the test will be reported as failing. For example a\nfailing integer check would look like\n\n # check \"x >= 3\" failed at my-test.c:102\n # left: 2\n # right: 3\n not ok 1 - x is greater than or equal to three\n\nThere are a number of check functions implemented so far. check() checks\na boolean condition, check_int(), check_uint() and check_char() take two\nvalues to compare and a comparison operator. check_str() will check if\ntwo strings are equal. Custom checks are simple to implement as shown in\nthe comments above test_assert() in test-lib.h.\n\nTests can be skipped with test_skip() which can be supplied with a\nreason for skipping which it will print. Tests can print diagnostic\nmessages with test_msg(). Checks that are known to fail can be wrapped\nin TEST_TODO().\n\nThere are a couple of example test programs included in this\npatch. t-basic.c implements some self-tests and demonstrates the\ndiagnostic output for failing test. The output of this program is\nchecked by t0080-unit-test-output.sh. t-strbuf.c shows some example\nunit tests for strbuf.c\n\nThe unit tests will be built as part of the default \"make all\" target,\nto avoid bitrot. If you wish to build just the unit tests, you can run\n\"make build-unit-tests\". To run the tests, you can use \"make unit-tests\"\nor run the test binaries directly, as in \"./t/unit-tests/bin/t-strbuf\".\n\nSigned-off-by: Phillip Wood \nSigned-off-by: Josh Steadmon \n---\n Makefile | 28 ++-\n t/Makefile | 15 +-\n t/t0080-unit-test-output.sh | 58 +++++++\n t/unit-tests/.gitignore | 1 +\n t/unit-tests/t-basic.c | 95 +++++++++++\n t/unit-tests/t-strbuf.c | 120 +++++++++++++\n t/unit-tests/test-lib.c | 330 ++++++++++++++++++++++++++++++++++++\n t/unit-tests/test-lib.h | 149 ++++++++++++++++\n 8 files changed, 792 insertions(+), 4 deletions(-)\n create mode 100755 t/t0080-unit-test-output.sh\n create mode 100644 t/unit-tests/.gitignore\n create mode 100644 t/unit-tests/t-basic.c\n create mode 100644 t/unit-tests/t-strbuf.c\n create mode 100644 t/unit-tests/test-lib.c\n create mode 100644 t/unit-tests/test-lib.h\n\ndiff --git a/Makefile b/Makefile\nindex e440728c24..18c13f06c0 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -682,6 +682,9 @@ TEST_BUILTINS_OBJS =\n TEST_OBJS =\n TEST_PROGRAMS_NEED_X =\n THIRD_PARTY_SOURCES =\n+UNIT_TEST_PROGRAMS =\n+UNIT_TEST_DIR = t/unit-tests\n+UNIT_TEST_BIN = $(UNIT_TEST_DIR)/bin\n \n # Having this variable in your environment would break pipelines because\n # you cause \"cd\" to echo its destination to stdout. It can also take\n@@ -1331,6 +1334,12 @@ THIRD_PARTY_SOURCES += compat/regex/%\n THIRD_PARTY_SOURCES += sha1collisiondetection/%\n THIRD_PARTY_SOURCES += sha1dc/%\n \n+UNIT_TEST_PROGRAMS += t-basic\n+UNIT_TEST_PROGRAMS += t-strbuf\n+UNIT_TEST_PROGS = $(patsubst %,$(UNIT_TEST_BIN)/%$X,$(UNIT_TEST_PROGRAMS))\n+UNIT_TEST_OBJS = $(patsubst %,$(UNIT_TEST_DIR)/%.o,$(UNIT_TEST_PROGRAMS))\n+UNIT_TEST_OBJS += $(UNIT_TEST_DIR)/test-lib.o\n+\n # xdiff and reftable libs may in turn depend on what is in libgit.a\n GITLIBS = common-main.o $(LIB_FILE) $(XDIFF_LIB) $(REFTABLE_LIB) $(LIB_FILE)\n EXTLIBS =\n@@ -2672,6 +2681,7 @@ OBJECTS += $(TEST_OBJS)\n OBJECTS += $(XDIFF_OBJS)\n OBJECTS += $(FUZZ_OBJS)\n OBJECTS += $(REFTABLE_OBJS) $(REFTABLE_TEST_OBJS)\n+OBJECTS += $(UNIT_TEST_OBJS)\n \n ifndef NO_CURL\n \tOBJECTS += http.o http-walker.o remote-curl.o\n@@ -3167,7 +3177,7 @@ endif\n \n test_bindir_programs := $(patsubst %,bin-wrappers/%,$(BINDIR_PROGRAMS_NEED_X) $(BINDIR_PROGRAMS_NO_X) $(TEST_PROGRAMS_NEED_X))\n \n-all:: $(TEST_PROGRAMS) $(test_bindir_programs)\n+all:: $(TEST_PROGRAMS) $(test_bindir_programs) $(UNIT_TEST_PROGS)\n \n bin-wrappers/%: wrap-for-bin.sh\n \t$(call mkdir_p_parent_template)\n@@ -3592,7 +3602,7 @@ endif\n \n artifacts-tar:: $(ALL_COMMANDS_TO_INSTALL) $(SCRIPT_LIB) $(OTHER_PROGRAMS) \\\n \t\tGIT-BUILD-OPTIONS $(TEST_PROGRAMS) $(test_bindir_programs) \\\n-\t\t$(MOFILES)\n+\t\t$(UNIT_TEST_PROGS) $(MOFILES)\n \t$(QUIET_SUBDIR0)templates $(QUIET_SUBDIR1) \\\n \t\tSHELL_PATH='$(SHELL_PATH_SQ)' PERL_PATH='$(PERL_PATH_SQ)'\n \ttest -n \"$(ARTIFACTS_DIRECTORY)\"\n@@ -3653,7 +3663,7 @@ clean: profile-clean coverage-clean cocciclean\n \t$(RM) $(OBJECTS)\n \t$(RM) $(LIB_FILE) $(XDIFF_LIB) $(REFTABLE_LIB) $(REFTABLE_TEST_LIB)\n \t$(RM) $(ALL_PROGRAMS) $(SCRIPT_LIB) $(BUILT_INS) $(OTHER_PROGRAMS)\n-\t$(RM) $(TEST_PROGRAMS)\n+\t$(RM) $(TEST_PROGRAMS) $(UNIT_TEST_PROGS)\n \t$(RM) $(FUZZ_PROGRAMS)\n \t$(RM) $(SP_OBJ)\n \t$(RM) $(HCC)\n@@ -3831,3 +3841,15 @@ $(FUZZ_PROGRAMS): all\n \t\t$(XDIFF_OBJS) $(EXTLIBS) git.o $@.o $(LIB_FUZZING_ENGINE) -o $@\n \n fuzz-all: $(FUZZ_PROGRAMS)\n+\n+$(UNIT_TEST_BIN):\n+\t@mkdir -p $(UNIT_TEST_BIN)\n+\n+$(UNIT_TEST_PROGS): $(UNIT_TEST_BIN)/%$X: $(UNIT_TEST_DIR)/%.o $(UNIT_TEST_DIR)/test-lib.o $(GITLIBS) GIT-LDFLAGS $(UNIT_TEST_BIN)\n+\t$(QUIET_LINK)$(CC) $(ALL_CFLAGS) -o $@ $(ALL_LDFLAGS) \\\n+\t\t$(filter %.o,$^) $(filter %.a,$^) $(LIBS)\n+\n+.PHONY: build-unit-tests unit-tests\n+build-unit-tests: $(UNIT_TEST_PROGS)\n+unit-tests: $(UNIT_TEST_PROGS)\n+\t$(MAKE) -C t/ unit-tests\ndiff --git a/t/Makefile b/t/Makefile\nindex 3e00cdd801..75d9330437 100644\n--- a/t/Makefile\n+++ b/t/Makefile\n@@ -17,6 +17,7 @@ TAR ?= $(TAR)\n RM ?= rm -f\n PROVE ?= prove\n DEFAULT_TEST_TARGET ?= test\n+DEFAULT_UNIT_TEST_TARGET ?= unit-tests-raw\n TEST_LINT ?= test-lint\n \n ifdef TEST_OUTPUT_DIRECTORY\n@@ -41,6 +42,7 @@ TPERF = $(sort $(wildcard perf/p[0-9][0-9][0-9][0-9]-*.sh))\n TINTEROP = $(sort $(wildcard interop/i[0-9][0-9][0-9][0-9]-*.sh))\n CHAINLINTTESTS = $(sort $(patsubst chainlint/%.test,%,$(wildcard chainlint/*.test)))\n CHAINLINT = '$(PERL_PATH_SQ)' chainlint.pl\n+UNIT_TESTS = $(sort $(filter-out unit-tests/bin/t-basic%,$(wildcard unit-tests/bin/t-*)))\n \n # `test-chainlint` (which is a dependency of `test-lint`, `test` and `prove`)\n # checks all tests in all scripts via a single invocation, so tell individual\n@@ -65,6 +67,17 @@ prove: pre-clean check-chainlint $(TEST_LINT)\n $(T):\n \t@echo \"*** $@ ***\"; '$(TEST_SHELL_PATH_SQ)' $@ $(GIT_TEST_OPTS)\n \n+$(UNIT_TESTS):\n+\t@echo \"*** $@ ***\"; $@\n+\n+.PHONY: unit-tests unit-tests-raw unit-tests-prove\n+unit-tests: $(DEFAULT_UNIT_TEST_TARGET)\n+\n+unit-tests-raw: $(UNIT_TESTS)\n+\n+unit-tests-prove:\n+\t@echo \"*** prove - unit tests ***\"; $(PROVE) $(GIT_PROVE_OPTS) $(UNIT_TESTS)\n+\n pre-clean:\n \t$(RM) -r '$(TEST_RESULTS_DIRECTORY_SQ)'\n \n@@ -149,4 +162,4 @@ perf:\n \t$(MAKE) -C perf/ all\n \n .PHONY: pre-clean $(T) aggregate-results clean valgrind perf \\\n-\tcheck-chainlint clean-chainlint test-chainlint\n+\tcheck-chainlint clean-chainlint test-chainlint $(UNIT_TESTS)\ndiff --git a/t/t0080-unit-test-output.sh b/t/t0080-unit-test-output.sh\nnew file mode 100755\nindex 0000000000..961b54b06c\n--- /dev/null\n+++ b/t/t0080-unit-test-output.sh\n@@ -0,0 +1,58 @@\n+#!/bin/sh\n+\n+test_description='Test the output of the unit test framework'\n+\n+. ./test-lib.sh\n+\n+test_expect_success 'TAP output from unit tests' '\n+\tcat >expect <<-EOF &&\n+\tok 1 - passing test\n+\tok 2 - passing test and assertion return 1\n+\t# check \"1 == 2\" failed at t/unit-tests/t-basic.c:76\n+\t# left: 1\n+\t# right: 2\n+\tnot ok 3 - failing test\n+\tok 4 - failing test and assertion return 0\n+\tnot ok 5 - passing TEST_TODO() # TODO\n+\tok 6 - passing TEST_TODO() returns 1\n+\t# todo check ${SQ}check(x)${SQ} succeeded at t/unit-tests/t-basic.c:25\n+\tnot ok 7 - failing TEST_TODO()\n+\tok 8 - failing TEST_TODO() returns 0\n+\t# check \"0\" failed at t/unit-tests/t-basic.c:30\n+\t# skipping test - missing prerequisite\n+\t# skipping check ${SQ}1${SQ} at t/unit-tests/t-basic.c:32\n+\tok 9 - test_skip() # SKIP\n+\tok 10 - skipped test returns 1\n+\t# skipping test - missing prerequisite\n+\tok 11 - test_skip() inside TEST_TODO() # SKIP\n+\tok 12 - test_skip() inside TEST_TODO() returns 1\n+\t# check \"0\" failed at t/unit-tests/t-basic.c:48\n+\tnot ok 13 - TEST_TODO() after failing check\n+\tok 14 - TEST_TODO() after failing check returns 0\n+\t# check \"0\" failed at t/unit-tests/t-basic.c:56\n+\tnot ok 15 - failing check after TEST_TODO()\n+\tok 16 - failing check after TEST_TODO() returns 0\n+\t# check \"!strcmp(\"\\thello\\\\\\\\\", \"there\\\"\\n\")\" failed at t/unit-tests/t-basic.c:61\n+\t# left: \"\\011hello\\\\\\\\\"\n+\t# right: \"there\\\"\\012\"\n+\t# check \"!strcmp(\"NULL\", NULL)\" failed at t/unit-tests/t-basic.c:62\n+\t# left: \"NULL\"\n+\t# right: NULL\n+\t# check \"${SQ}a${SQ} == ${SQ}\\n${SQ}\" failed at t/unit-tests/t-basic.c:63\n+\t# left: ${SQ}a${SQ}\n+\t# right: ${SQ}\\012${SQ}\n+\t# check \"${SQ}\\\\\\\\${SQ} == ${SQ}\\\\${SQ}${SQ}\" failed at t/unit-tests/t-basic.c:64\n+\t# left: ${SQ}\\\\\\\\${SQ}\n+\t# right: ${SQ}\\\\${SQ}${SQ}\n+\tnot ok 17 - messages from failing string and char comparison\n+\t# BUG: test has no checks at t/unit-tests/t-basic.c:91\n+\tnot ok 18 - test with no checks\n+\tok 19 - test with no checks returns 0\n+\t1..19\n+\tEOF\n+\n+\t! \"$GIT_BUILD_DIR\"/t/unit-tests/bin/t-basic >actual &&\n+\ttest_cmp expect actual\n+'\n+\n+test_done\ndiff --git a/t/unit-tests/.gitignore b/t/unit-tests/.gitignore\nnew file mode 100644\nindex 0000000000..5e56e040ec\n--- /dev/null\n+++ b/t/unit-tests/.gitignore\n@@ -0,0 +1 @@\n+/bin\ndiff --git a/t/unit-tests/t-basic.c b/t/unit-tests/t-basic.c\nnew file mode 100644\nindex 0000000000..fda1ae59a6\n--- /dev/null\n+++ b/t/unit-tests/t-basic.c\n@@ -0,0 +1,95 @@\n+#include \"test-lib.h\"\n+\n+/*\n+ * The purpose of this \"unit test\" is to verify a few invariants of the unit\n+ * test framework itself, as well as to provide examples of output from actually\n+ * failing tests. As such, it is intended that this test fails, and thus it\n+ * should not be run as part of `make unit-tests`. Instead, we verify it behaves\n+ * as expected in the integration test t0080-unit-test-output.sh\n+ */\n+\n+/* Used to store the return value of check_int(). */\n+static int check_res;\n+\n+/* Used to store the return value of TEST(). */\n+static int test_res;\n+\n+static void t_res(int expect)\n+{\n+\tcheck_int(check_res, ==, expect);\n+\tcheck_int(test_res, ==, expect);\n+}\n+\n+static void t_todo(int x)\n+{\n+\tcheck_res = TEST_TODO(check(x));\n+}\n+\n+static void t_skip(void)\n+{\n+\tcheck(0);\n+\ttest_skip(\"missing prerequisite\");\n+\tcheck(1);\n+}\n+\n+static int do_skip(void)\n+{\n+\ttest_skip(\"missing prerequisite\");\n+\treturn 1;\n+}\n+\n+static void t_skip_todo(void)\n+{\n+\tcheck_res = TEST_TODO(do_skip());\n+}\n+\n+static void t_todo_after_fail(void)\n+{\n+\tcheck(0);\n+\tTEST_TODO(check(0));\n+}\n+\n+static void t_fail_after_todo(void)\n+{\n+\tcheck(1);\n+\tTEST_TODO(check(0));\n+\tcheck(0);\n+}\n+\n+static void t_messages(void)\n+{\n+\tcheck_str(\"\\thello\\\\\", \"there\\\"\\n\");\n+\tcheck_str(\"NULL\", NULL);\n+\tcheck_char('a', ==, '\\n');\n+\tcheck_char('\\\\', ==, '\\'');\n+}\n+\n+static void t_empty(void)\n+{\n+\t; /* empty */\n+}\n+\n+int cmd_main(int argc, const char **argv)\n+{\n+\ttest_res = TEST(check_res = check_int(1, ==, 1), \"passing test\");\n+\tTEST(t_res(1), \"passing test and assertion return 1\");\n+\ttest_res = TEST(check_res = check_int(1, ==, 2), \"failing test\");\n+\tTEST(t_res(0), \"failing test and assertion return 0\");\n+\ttest_res = TEST(t_todo(0), \"passing TEST_TODO()\");\n+\tTEST(t_res(1), \"passing TEST_TODO() returns 1\");\n+\ttest_res = TEST(t_todo(1), \"failing TEST_TODO()\");\n+\tTEST(t_res(0), \"failing TEST_TODO() returns 0\");\n+\ttest_res = TEST(t_skip(), \"test_skip()\");\n+\tTEST(check_int(test_res, ==, 1), \"skipped test returns 1\");\n+\ttest_res = TEST(t_skip_todo(), \"test_skip() inside TEST_TODO()\");\n+\tTEST(t_res(1), \"test_skip() inside TEST_TODO() returns 1\");\n+\ttest_res = TEST(t_todo_after_fail(), \"TEST_TODO() after failing check\");\n+\tTEST(check_int(test_res, ==, 0), \"TEST_TODO() after failing check returns 0\");\n+\ttest_res = TEST(t_fail_after_todo(), \"failing check after TEST_TODO()\");\n+\tTEST(check_int(test_res, ==, 0), \"failing check after TEST_TODO() returns 0\");\n+\tTEST(t_messages(), \"messages from failing string and char comparison\");\n+\ttest_res = TEST(t_empty(), \"test with no checks\");\n+\tTEST(check_int(test_res, ==, 0), \"test with no checks returns 0\");\n+\n+\treturn test_done();\n+}\ndiff --git a/t/unit-tests/t-strbuf.c b/t/unit-tests/t-strbuf.c\nnew file mode 100644\nindex 0000000000..de434a4441\n--- /dev/null\n+++ b/t/unit-tests/t-strbuf.c\n@@ -0,0 +1,120 @@\n+#include \"test-lib.h\"\n+#include \"strbuf.h\"\n+\n+/* wrapper that supplies tests with an empty, initialized strbuf */\n+static void setup(void (*f)(struct strbuf*, void*), void *data)\n+{\n+\tstruct strbuf buf = STRBUF_INIT;\n+\n+\tf(&buf, data);\n+\tstrbuf_release(&buf);\n+\tcheck_uint(buf.len, ==, 0);\n+\tcheck_uint(buf.alloc, ==, 0);\n+}\n+\n+/* wrapper that supplies tests with a populated, initialized strbuf */\n+static void setup_populated(void (*f)(struct strbuf*, void*), char *init_str, void *data)\n+{\n+\tstruct strbuf buf = STRBUF_INIT;\n+\n+\tstrbuf_addstr(&buf, init_str);\n+\tcheck_uint(buf.len, ==, strlen(init_str));\n+\tf(&buf, data);\n+\tstrbuf_release(&buf);\n+\tcheck_uint(buf.len, ==, 0);\n+\tcheck_uint(buf.alloc, ==, 0);\n+}\n+\n+static int assert_sane_strbuf(struct strbuf *buf)\n+{\n+\t/* Initialized strbufs should always have a non-NULL buffer */\n+\tif (!check(!!buf->buf))\n+\t\treturn 0;\n+\t/* Buffers should always be NUL-terminated */\n+\tif (!check_char(buf->buf[buf->len], ==, '\\0'))\n+\t\treturn 0;\n+\t/*\n+\t * Freshly-initialized strbufs may not have a dynamically allocated\n+\t * buffer\n+\t */\n+\tif (buf->len == 0 && buf->alloc == 0)\n+\t\treturn 1;\n+\t/* alloc must be at least one byte larger than len */\n+\treturn check_uint(buf->len, <, buf->alloc);\n+}\n+\n+static void t_static_init(void)\n+{\n+\tstruct strbuf buf = STRBUF_INIT;\n+\n+\tcheck_uint(buf.len, ==, 0);\n+\tcheck_uint(buf.alloc, ==, 0);\n+\tcheck_char(buf.buf[0], ==, '\\0');\n+}\n+\n+static void t_dynamic_init(void)\n+{\n+\tstruct strbuf buf;\n+\n+\tstrbuf_init(&buf, 1024);\n+\tcheck(assert_sane_strbuf(&buf));\n+\tcheck_uint(buf.len, ==, 0);\n+\tcheck_uint(buf.alloc, >=, 1024);\n+\tcheck_char(buf.buf[0], ==, '\\0');\n+\tstrbuf_release(&buf);\n+}\n+\n+static void t_addch(struct strbuf *buf, void *data)\n+{\n+\tconst char *p_ch = data;\n+\tconst char ch = *p_ch;\n+\tsize_t orig_alloc = buf->alloc;\n+\tsize_t orig_len = buf->len;\n+\n+\tif (!check(assert_sane_strbuf(buf)))\n+\t\treturn;\n+\tstrbuf_addch(buf, ch);\n+\tif (!check(assert_sane_strbuf(buf)))\n+\t\treturn;\n+\tif (!(check_uint(buf->len, ==, orig_len + 1) &&\n+\t check_uint(buf->alloc, >=, orig_alloc)))\n+\t\treturn; /* avoid de-referencing buf->buf */\n+\tcheck_char(buf->buf[buf->len - 1], ==, ch);\n+\tcheck_char(buf->buf[buf->len], ==, '\\0');\n+}\n+\n+static void t_addstr(struct strbuf *buf, void *data)\n+{\n+\tconst char *text = data;\n+\tsize_t len = strlen(text);\n+\tsize_t orig_alloc = buf->alloc;\n+\tsize_t orig_len = buf->len;\n+\n+\tif (!check(assert_sane_strbuf(buf)))\n+\t\treturn;\n+\tstrbuf_addstr(buf, text);\n+\tif (!check(assert_sane_strbuf(buf)))\n+\t\treturn;\n+\tif (!(check_uint(buf->len, ==, orig_len + len) &&\n+\t check_uint(buf->alloc, >=, orig_alloc) &&\n+\t check_uint(buf->alloc, >, orig_len + len) &&\n+\t check_char(buf->buf[orig_len + len], ==, '\\0')))\n+\t return;\n+\tcheck_str(buf->buf + orig_len, text);\n+}\n+\n+int cmd_main(int argc, const char **argv)\n+{\n+\tif (!TEST(t_static_init(), \"static initialization works\"))\n+\t\ttest_skip_all(\"STRBUF_INIT is broken\");\n+\tTEST(t_dynamic_init(), \"dynamic initialization works\");\n+\tTEST(setup(t_addch, \"a\"), \"strbuf_addch adds char\");\n+\tTEST(setup(t_addch, \"\"), \"strbuf_addch adds NUL char\");\n+\tTEST(setup_populated(t_addch, \"initial value\", \"a\"),\n+\t \"strbuf_addch appends to initial value\");\n+\tTEST(setup(t_addstr, \"hello there\"), \"strbuf_addstr adds string\");\n+\tTEST(setup_populated(t_addstr, \"initial value\", \"hello there\"),\n+\t \"strbuf_addstr appends string to initial value\");\n+\n+\treturn test_done();\n+}\ndiff --git a/t/unit-tests/test-lib.c b/t/unit-tests/test-lib.c\nnew file mode 100644\nindex 0000000000..a2cc21c706\n--- /dev/null\n+++ b/t/unit-tests/test-lib.c\n@@ -0,0 +1,330 @@\n+#include \"test-lib.h\"\n+\n+enum result {\n+\tRESULT_NONE,\n+\tRESULT_FAILURE,\n+\tRESULT_SKIP,\n+\tRESULT_SUCCESS,\n+\tRESULT_TODO\n+};\n+\n+static struct {\n+\tenum result result;\n+\tint count;\n+\tunsigned failed :1;\n+\tunsigned lazy_plan :1;\n+\tunsigned running :1;\n+\tunsigned skip_all :1;\n+\tunsigned todo :1;\n+} ctx = {\n+\t.lazy_plan = 1,\n+\t.result = RESULT_NONE,\n+};\n+\n+static void msg_with_prefix(const char *prefix, const char *format, va_list ap)\n+{\n+\tfflush(stderr);\n+\tif (prefix)\n+\t\tfprintf(stdout, \"%s\", prefix);\n+\tvprintf(format, ap); /* TODO: handle newlines */\n+\tputc('\\n', stdout);\n+\tfflush(stdout);\n+}\n+\n+void test_msg(const char *format, ...)\n+{\n+\tva_list ap;\n+\n+\tva_start(ap, format);\n+\tmsg_with_prefix(\"# \", format, ap);\n+\tva_end(ap);\n+}\n+\n+void test_plan(int count)\n+{\n+\tassert(!ctx.running);\n+\n+\tfflush(stderr);\n+\tprintf(\"1..%d\\n\", count);\n+\tfflush(stdout);\n+\tctx.lazy_plan = 0;\n+}\n+\n+int test_done(void)\n+{\n+\tassert(!ctx.running);\n+\n+\tif (ctx.lazy_plan)\n+\t\ttest_plan(ctx.count);\n+\n+\treturn ctx.failed;\n+}\n+\n+void test_skip(const char *format, ...)\n+{\n+\tva_list ap;\n+\n+\tassert(ctx.running);\n+\n+\tctx.result = RESULT_SKIP;\n+\tva_start(ap, format);\n+\tif (format)\n+\t\tmsg_with_prefix(\"# skipping test - \", format, ap);\n+\tva_end(ap);\n+}\n+\n+void test_skip_all(const char *format, ...)\n+{\n+\tva_list ap;\n+\tconst char *prefix;\n+\n+\tif (!ctx.count && ctx.lazy_plan) {\n+\t\t/* We have not printed a test plan yet */\n+\t\tprefix = \"1..0 # SKIP \";\n+\t\tctx.lazy_plan = 0;\n+\t} else {\n+\t\t/* We have already printed a test plan */\n+\t\tprefix = \"Bail out! # \";\n+\t\tctx.failed = 1;\n+\t}\n+\tctx.skip_all = 1;\n+\tctx.result = RESULT_SKIP;\n+\tva_start(ap, format);\n+\tmsg_with_prefix(prefix, format, ap);\n+\tva_end(ap);\n+}\n+\n+int test__run_begin(void)\n+{\n+\tassert(!ctx.running);\n+\n+\tctx.count++;\n+\tctx.result = RESULT_NONE;\n+\tctx.running = 1;\n+\n+\treturn ctx.skip_all;\n+}\n+\n+static void print_description(const char *format, va_list ap)\n+{\n+\tif (format) {\n+\t\tfputs(\" - \", stdout);\n+\t\tvprintf(format, ap);\n+\t}\n+}\n+\n+int test__run_end(int was_run UNUSED, const char *location, const char *format, ...)\n+{\n+\tva_list ap;\n+\n+\tassert(ctx.running);\n+\tassert(!ctx.todo);\n+\n+\tfflush(stderr);\n+\tva_start(ap, format);\n+\tif (!ctx.skip_all) {\n+\t\tswitch (ctx.result) {\n+\t\tcase RESULT_SUCCESS:\n+\t\t\tprintf(\"ok %d\", ctx.count);\n+\t\t\tprint_description(format, ap);\n+\t\t\tbreak;\n+\n+\t\tcase RESULT_FAILURE:\n+\t\t\tprintf(\"not ok %d\", ctx.count);\n+\t\t\tprint_description(format, ap);\n+\t\t\tbreak;\n+\n+\t\tcase RESULT_TODO:\n+\t\t\tprintf(\"not ok %d\", ctx.count);\n+\t\t\tprint_description(format, ap);\n+\t\t\tprintf(\" # TODO\");\n+\t\t\tbreak;\n+\n+\t\tcase RESULT_SKIP:\n+\t\t\tprintf(\"ok %d\", ctx.count);\n+\t\t\tprint_description(format, ap);\n+\t\t\tprintf(\" # SKIP\");\n+\t\t\tbreak;\n+\n+\t\tcase RESULT_NONE:\n+\t\t\ttest_msg(\"BUG: test has no checks at %s\", location);\n+\t\t\tprintf(\"not ok %d\", ctx.count);\n+\t\t\tprint_description(format, ap);\n+\t\t\tctx.result = RESULT_FAILURE;\n+\t\t\tbreak;\n+\t\t}\n+\t}\n+\tva_end(ap);\n+\tctx.running = 0;\n+\tif (ctx.skip_all)\n+\t\treturn 1;\n+\tputc('\\n', stdout);\n+\tfflush(stdout);\n+\tctx.failed |= ctx.result == RESULT_FAILURE;\n+\n+\treturn ctx.result != RESULT_FAILURE;\n+}\n+\n+static void test_fail(void)\n+{\n+\tassert(ctx.result != RESULT_SKIP);\n+\n+\tctx.result = RESULT_FAILURE;\n+}\n+\n+static void test_pass(void)\n+{\n+\tassert(ctx.result != RESULT_SKIP);\n+\n+\tif (ctx.result == RESULT_NONE)\n+\t\tctx.result = RESULT_SUCCESS;\n+}\n+\n+static void test_todo(void)\n+{\n+\tassert(ctx.result != RESULT_SKIP);\n+\n+\tif (ctx.result != RESULT_FAILURE)\n+\t\tctx.result = RESULT_TODO;\n+}\n+\n+int test_assert(const char *location, const char *check, int ok)\n+{\n+\tassert(ctx.running);\n+\n+\tif (ctx.result == RESULT_SKIP) {\n+\t\ttest_msg(\"skipping check '%s' at %s\", check, location);\n+\t\treturn 1;\n+\t}\n+\tif (!ctx.todo) {\n+\t\tif (ok) {\n+\t\t\ttest_pass();\n+\t\t} else {\n+\t\t\ttest_msg(\"check \\\"%s\\\" failed at %s\", check, location);\n+\t\t\ttest_fail();\n+\t\t}\n+\t}\n+\n+\treturn !!ok;\n+}\n+\n+void test__todo_begin(void)\n+{\n+\tassert(ctx.running);\n+\tassert(!ctx.todo);\n+\n+\tctx.todo = 1;\n+}\n+\n+int test__todo_end(const char *location, const char *check, int res)\n+{\n+\tassert(ctx.running);\n+\tassert(ctx.todo);\n+\n+\tctx.todo = 0;\n+\tif (ctx.result == RESULT_SKIP)\n+\t\treturn 1;\n+\tif (res) {\n+\t\ttest_msg(\"todo check '%s' succeeded at %s\", check, location);\n+\t\ttest_fail();\n+\t} else {\n+\t\ttest_todo();\n+\t}\n+\n+\treturn !res;\n+}\n+\n+int check_bool_loc(const char *loc, const char *check, int ok)\n+{\n+\treturn test_assert(loc, check, ok);\n+}\n+\n+union test__tmp test__tmp[2];\n+\n+int check_int_loc(const char *loc, const char *check, int ok,\n+\t\t intmax_t a, intmax_t b)\n+{\n+\tint ret = test_assert(loc, check, ok);\n+\n+\tif (!ret) {\n+\t\ttest_msg(\" left: %\"PRIdMAX, a);\n+\t\ttest_msg(\" right: %\"PRIdMAX, b);\n+\t}\n+\n+\treturn ret;\n+}\n+\n+int check_uint_loc(const char *loc, const char *check, int ok,\n+\t\t uintmax_t a, uintmax_t b)\n+{\n+\tint ret = test_assert(loc, check, ok);\n+\n+\tif (!ret) {\n+\t\ttest_msg(\" left: %\"PRIuMAX, a);\n+\t\ttest_msg(\" right: %\"PRIuMAX, b);\n+\t}\n+\n+\treturn ret;\n+}\n+\n+static void print_one_char(char ch, char quote)\n+{\n+\tif ((unsigned char)ch < 0x20u || ch == 0x7f) {\n+\t\t/* TODO: improve handling of \\a, \\b, \\f ... */\n+\t\tprintf(\"\\\\%03o\", (unsigned char)ch);\n+\t} else {\n+\t\tif (ch == '\\\\' || ch == quote)\n+\t\t\tputc('\\\\', stdout);\n+\t\tputc(ch, stdout);\n+\t}\n+}\n+\n+static void print_char(const char *prefix, char ch)\n+{\n+\tprintf(\"# %s: '\", prefix);\n+\tprint_one_char(ch, '\\'');\n+\tfputs(\"'\\n\", stdout);\n+}\n+\n+int check_char_loc(const char *loc, const char *check, int ok, char a, char b)\n+{\n+\tint ret = test_assert(loc, check, ok);\n+\n+\tif (!ret) {\n+\t\tfflush(stderr);\n+\t\tprint_char(\" left\", a);\n+\t\tprint_char(\" right\", b);\n+\t\tfflush(stdout);\n+\t}\n+\n+\treturn ret;\n+}\n+\n+static void print_str(const char *prefix, const char *str)\n+{\n+\tprintf(\"# %s: \", prefix);\n+\tif (!str) {\n+\t\tfputs(\"NULL\\n\", stdout);\n+\t} else {\n+\t\tputc('\"', stdout);\n+\t\twhile (*str)\n+\t\t\tprint_one_char(*str++, '\"');\n+\t\tfputs(\"\\\"\\n\", stdout);\n+\t}\n+}\n+\n+int check_str_loc(const char *loc, const char *check,\n+\t\t const char *a, const char *b)\n+{\n+\tint ok = (!a && !b) || (a && b && !strcmp(a, b));\n+\tint ret = test_assert(loc, check, ok);\n+\n+\tif (!ret) {\n+\t\tfflush(stderr);\n+\t\tprint_str(\" left\", a);\n+\t\tprint_str(\" right\", b);\n+\t\tfflush(stdout);\n+\t}\n+\n+\treturn ret;\n+}\ndiff --git a/t/unit-tests/test-lib.h b/t/unit-tests/test-lib.h\nnew file mode 100644\nindex 0000000000..a8f07ae0b7\n--- /dev/null\n+++ b/t/unit-tests/test-lib.h\n@@ -0,0 +1,149 @@\n+#ifndef TEST_LIB_H\n+#define TEST_LIB_H\n+\n+#include \"git-compat-util.h\"\n+\n+/*\n+ * Run a test function, returns 1 if the test succeeds, 0 if it\n+ * fails. If test_skip_all() has been called then the test will not be\n+ * run. The description for each test should be unique. For example:\n+ *\n+ * TEST(test_something(arg1, arg2), \"something %d %d\", arg1, arg2)\n+ */\n+#define TEST(t, ...)\t\t\t\t\t\\\n+\ttest__run_end(test__run_begin() ? 0 : (t, 1),\t\\\n+\t\t TEST_LOCATION(), __VA_ARGS__)\n+\n+/*\n+ * Print a test plan, should be called before any tests. If the number\n+ * of tests is not known in advance test_done() will automatically\n+ * print a plan at the end of the test program.\n+ */\n+void test_plan(int count);\n+\n+/*\n+ * test_done() must be called at the end of main(). It will print the\n+ * plan if plan() was not called at the beginning of the test program\n+ * and returns the exit code for the test program.\n+ */\n+int test_done(void);\n+\n+/* Skip the current test. */\n+__attribute__((format (printf, 1, 2)))\n+void test_skip(const char *format, ...);\n+\n+/* Skip all remaining tests. */\n+__attribute__((format (printf, 1, 2)))\n+void test_skip_all(const char *format, ...);\n+\n+/* Print a diagnostic message to stdout. */\n+__attribute__((format (printf, 1, 2)))\n+void test_msg(const char *format, ...);\n+\n+/*\n+ * Test checks are built around test_assert(). checks return 1 on\n+ * success, 0 on failure. If any check fails then the test will fail. To\n+ * create a custom check define a function that wraps test_assert() and\n+ * a macro to wrap that function to provide a source location and\n+ * stringified arguments. Custom checks that take pointer arguments\n+ * should be careful to check that they are non-NULL before\n+ * dereferencing them. For example:\n+ *\n+ * static int check_oid_loc(const char *loc, const char *check,\n+ *\t\t\t struct object_id *a, struct object_id *b)\n+ * {\n+ *\t int res = test_assert(loc, check, a && b && oideq(a, b));\n+ *\n+ *\t if (!res) {\n+ *\t\t test_msg(\" left: %s\", a ? oid_to_hex(a) : \"NULL\";\n+ *\t\t test_msg(\" right: %s\", b ? oid_to_hex(a) : \"NULL\";\n+ *\n+ *\t }\n+ *\t return res;\n+ * }\n+ *\n+ * #define check_oid(a, b) \\\n+ *\t check_oid_loc(TEST_LOCATION(), \"oideq(\"#a\", \"#b\")\", a, b)\n+ */\n+int test_assert(const char *location, const char *check, int ok);\n+\n+/* Helper macro to pass the location to checks */\n+#define TEST_LOCATION() TEST__MAKE_LOCATION(__LINE__)\n+\n+/* Check a boolean condition. */\n+#define check(x)\t\t\t\t\\\n+\tcheck_bool_loc(TEST_LOCATION(), #x, x)\n+int check_bool_loc(const char *loc, const char *check, int ok);\n+\n+/*\n+ * Compare two integers. Prints a message with the two values if the\n+ * comparison fails. NB this is not thread safe.\n+ */\n+#define check_int(a, op, b)\t\t\t\t\t\t\\\n+\t(test__tmp[0].i = (a), test__tmp[1].i = (b),\t\t\t\\\n+\t check_int_loc(TEST_LOCATION(), #a\" \"#op\" \"#b,\t\t\t\\\n+\t\t test__tmp[0].i op test__tmp[1].i,\t\t\\\n+\t\t test__tmp[0].i, test__tmp[1].i))\n+int check_int_loc(const char *loc, const char *check, int ok,\n+\t\t intmax_t a, intmax_t b);\n+\n+/*\n+ * Compare two unsigned integers. Prints a message with the two values\n+ * if the comparison fails. NB this is not thread safe.\n+ */\n+#define check_uint(a, op, b)\t\t\t\t\t\t\\\n+\t(test__tmp[0].u = (a), test__tmp[1].u = (b),\t\t\t\\\n+\t check_uint_loc(TEST_LOCATION(), #a\" \"#op\" \"#b,\t\t\t\\\n+\t\t\ttest__tmp[0].u op test__tmp[1].u,\t\t\\\n+\t\t\ttest__tmp[0].u, test__tmp[1].u))\n+int check_uint_loc(const char *loc, const char *check, int ok,\n+\t\t uintmax_t a, uintmax_t b);\n+\n+/*\n+ * Compare two chars. Prints a message with the two values if the\n+ * comparison fails. NB this is not thread safe.\n+ */\n+#define check_char(a, op, b)\t\t\t\t\t\t\\\n+\t(test__tmp[0].c = (a), test__tmp[1].c = (b),\t\t\t\\\n+\t check_char_loc(TEST_LOCATION(), #a\" \"#op\" \"#b,\t\t\t\\\n+\t\t\ttest__tmp[0].c op test__tmp[1].c,\t\t\\\n+\t\t\ttest__tmp[0].c, test__tmp[1].c))\n+int check_char_loc(const char *loc, const char *check, int ok,\n+\t\t char a, char b);\n+\n+/* Check whether two strings are equal. */\n+#define check_str(a, b)\t\t\t\t\t\t\t\\\n+\tcheck_str_loc(TEST_LOCATION(), \"!strcmp(\"#a\", \"#b\")\", a, b)\n+int check_str_loc(const char *loc, const char *check,\n+\t\t const char *a, const char *b);\n+\n+/*\n+ * Wrap a check that is known to fail. If the check succeeds then the\n+ * test will fail. Returns 1 if the check fails, 0 if it\n+ * succeeds. For example:\n+ *\n+ * TEST_TODO(check(0));\n+ */\n+#define TEST_TODO(check) \\\n+\t(test__todo_begin(), test__todo_end(TEST_LOCATION(), #check, check))\n+\n+/* Private helpers */\n+\n+#define TEST__STR(x) #x\n+#define TEST__MAKE_LOCATION(line) __FILE__ \":\" TEST__STR(line)\n+\n+union test__tmp {\n+\tintmax_t i;\n+\tuintmax_t u;\n+\tchar c;\n+};\n+\n+extern union test__tmp test__tmp[2];\n+\n+int test__run_begin(void);\n+__attribute__((format (printf, 3, 4)))\n+int test__run_end(int, const char *, const char *, ...);\n+void test__todo_begin(void);\n+int test__todo_end(const char *, const char *, int);\n+\n+#endif /* TEST_LIB_H */\n-- \n2.42.0.869.gea05f2083d-goog\n\n"},{"id":"484658","messageId":"0129ec062c8f7ac2a87cc273454a1ca63705fc55.1699555664.git.steadmon@google.com","threadId":"59937","inReplyTo":"cover.1699555664.git.steadmon@google.com","subject":"[PATCH v10 3/3] ci: run unit tests in CI","fromName":"Josh Steadmon","fromEmail":"steadmon@google.com","sentAt":"2023-11-09T18:50:44Z","receivedAt":"2023-11-09T18:50:55Z","isPatch":true,"sender":{"key":"steadmon@google.com","avatar":"https://avatars.githubusercontent.com/u/2654920?v=4"},"body":"Run unit tests in both Cirrus and GitHub CI. For sharded CI instances\n(currently just Windows on GitHub), run only on the first shard. This is\nOK while we have only a single unit test executable, but we may wish to\ndistribute tests more evenly when we add new unit tests in the future.\n\nWe may also want to add more status output in our unit test framework,\nso that we can do similar post-processing as in\nci/lib.sh:handle_failed_tests().\n\nSigned-off-by: Josh Steadmon \n---\n .cirrus.yml | 2 +-\n ci/run-build-and-tests.sh | 2 ++\n ci/run-test-slice.sh | 5 +++++\n 3 files changed, 8 insertions(+), 1 deletion(-)\n\ndiff --git a/.cirrus.yml b/.cirrus.yml\nindex 4860bebd32..b6280692d2 100644\n--- a/.cirrus.yml\n+++ b/.cirrus.yml\n@@ -19,4 +19,4 @@ freebsd_12_task:\n build_script:\n - su git -c gmake\n test_script:\n- - su git -c 'gmake test'\n+ - su git -c 'gmake DEFAULT_UNIT_TEST_TARGET=unit-tests-prove test unit-tests'\ndiff --git a/ci/run-build-and-tests.sh b/ci/run-build-and-tests.sh\nindex 2528f25e31..7a1466b868 100755\n--- a/ci/run-build-and-tests.sh\n+++ b/ci/run-build-and-tests.sh\n@@ -50,6 +50,8 @@ if test -n \"$run_tests\"\n then\n \tgroup \"Run tests\" make test ||\n \thandle_failed_tests\n+\tgroup \"Run unit tests\" \\\n+\t\tmake DEFAULT_UNIT_TEST_TARGET=unit-tests-prove unit-tests\n fi\n check_unignored_build_artifacts\n \ndiff --git a/ci/run-test-slice.sh b/ci/run-test-slice.sh\nindex a3c67956a8..ae8094382f 100755\n--- a/ci/run-test-slice.sh\n+++ b/ci/run-test-slice.sh\n@@ -15,4 +15,9 @@ group \"Run tests\" make --quiet -C t T=\"$(cd t &&\n \ttr '\\n' ' ')\" ||\n handle_failed_tests\n \n+# We only have one unit test at the moment, so run it in the first slice\n+if [ \"$1\" == \"0\" ] ; then\n+\tgroup \"Run unit tests\" make --quiet -C t unit-tests-prove\n+fi\n+\n check_unignored_build_artifacts\n-- \n2.42.0.869.gea05f2083d-goog\n\n"},{"id":"484680","messageId":"xmqq1qcyzfgp.fsf@gitster.g","threadId":"59937","inReplyTo":"f706ba9b682e7c4070d49086ad3af582bf269c79.1699555664.git.steadmon@google.com","subject":"Re: [PATCH v10 1/3] unit tests: Add a project plan document","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2023-11-09T23:15:02Z","receivedAt":"2023-11-09T23:15:11Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Josh Steadmon writes:\n\n> In our current testing environment, we spend a significant amount of\n> effort crafting end-to-end tests for error conditions that could easily\n> be captured by unit tests (or we simply forgo some hard-to-setup and\n> rare error conditions). Describe what we hope to accomplish by\n> implementing unit tests, and explain some open questions and milestones.\n> Discuss desired features for test frameworks/harnesses, and provide a\n> comparison of several different frameworks. Finally, document our\n> rationale for implementing a custom framework.\n>\n> Co-authored-by: Calvin Wan \n> Signed-off-by: Calvin Wan \n> Signed-off-by: Josh Steadmon \n> ---\n> Documentation/Makefile | 1 +\n> Documentation/technical/unit-tests.txt | 240 +++++++++++++++++++++++++\n> 2 files changed, 241 insertions(+)\n> create mode 100644 Documentation/technical/unit-tests.txt\n\nLooks good. I'll downcase \"Add\" on the title to match what I have\nin my tree, but otherwise it looks OK to me. Let's see if we can\nmark this round ready for 'next' and check if we hear complaints.\n\nI have to make sure I do not forget about the other topic that\nbuilds on top of this one.\n\nThanks.\n\n"}]}