{"thread":{"id":"60754","subject":"[RFC PATCH 0/4] test-tool: add unit test suite runner","startedAt":"2024-01-16T22:23:04Z","lastAt":"2024-05-06T21:11:10Z","messageCount":99,"participants":["Josh Steadmon","Junio C Hamano","Jeff King","phillip.wood123@gmail.com","rsbecker@nexbridge.com"],"isPatch":true,"patchVersion":1,"patchTotal":4},"messages":[{"id":"486890","messageId":"cover.1705443632.git.steadmon@google.com","threadId":"60754","inReplyTo":null,"subject":"[RFC PATCH 0/4] test-tool: add unit test suite runner","fromName":"Josh Steadmon","fromEmail":"steadmon@google.com","sentAt":"2024-01-16T22:22:57Z","receivedAt":"2024-01-16T22:23:04Z","isPatch":true,"sender":{"key":"steadmon@google.com","avatar":"https://avatars.githubusercontent.com/u/2654920?v=4"},"body":"For various reasons (see discussion at [1]) we would like an alternative\nto `prove` for running test suites (including the unit tests) on\nWindows.\n\nThis series extends the existing `test-tool run-command testsuite` to\nsupport running unit tests. In addition, it includes some small\ncleanups: we move t-basic out of the unit-tests directory, and add a\ntest wrapper script to allow unit tests and the shell test suite to run\nin a single `prove` process.\n\nSome known remaining bits of work:\n* We need to filter out cmake *.pdb files when running with `test-tool`.\n* We should investigate switching the Windows CI to use `test-tool`\n instead of prove.\n* We should determine whether it is confusing or otherwise harmful to\n people's workflow to have the unit tests run in parallel with shell\n tests when using prove as the default test target.\n\n[1] https://lore.kernel.org/git/pull.1613.git.1699894837844.gitgitgadget@gmail.com/\n\n\nJeff King (1):\n t/Makefile: run unit tests alongside shell tests\n\nJosh Steadmon (3):\n t0080: turn t-basic unit test into a helper\n test-tool run-command testsuite: support unit tests\n unit tests: add rule for running with test-tool\n\n Makefile | 19 +++++++++++----\n t/Makefile | 13 +++++++---\n t/helper/test-run-command.c | 40 ++++++++++++++++++++++++-------\n t/run-test.sh | 13 ++++++++++\n t/t0080-unit-test-output.sh | 24 +++++++++----------\n t/t0080/.gitignore | 1 +\n t/{unit-tests => t0080}/t-basic.c | 2 +-\n 7 files changed, 83 insertions(+), 29 deletions(-)\n create mode 100755 t/run-test.sh\n create mode 100644 t/t0080/.gitignore\n rename t/{unit-tests => t0080}/t-basic.c (98%)\n\n\nbase-commit: a26002b62827b89a19b1084bd75d9371d565d03c\n-- \n2.43.0.381.gb435a96ce8-goog\n\n"},{"id":"486891","messageId":"a9f67ed703c8314f0f0507ffb83b503717b846b3.1705443632.git.steadmon@google.com","threadId":"60754","inReplyTo":"cover.1705443632.git.steadmon@google.com","subject":"[RFC PATCH 1/4] t0080: turn t-basic unit test into a helper","fromName":"Josh Steadmon","fromEmail":"steadmon@google.com","sentAt":"2024-01-16T22:22:58Z","receivedAt":"2024-01-16T22:23:06Z","isPatch":true,"sender":{"key":"steadmon@google.com","avatar":"https://avatars.githubusercontent.com/u/2654920?v=4"},"body":"While t/unit-tests/t-basic.c uses the unit-test framework added in\ne137fe3b29 (unit tests: add TAP unit test framework, 2023-11-09), it is\nnot a true unit test in that it intentionally fails in order to exercise\nvarious codepaths in the unit-test framework. Thus, we intentionally\nexclude it when running unit tests through the various t/Makefile\ntargets. Instead, it is executed by t0080-unit-test-output.sh, which\nverifies its output follows the TAP format expected for the various\npass, skip, or fail cases.\n\nAs such, it makes more sense for t-basic to be a helper item for\nt0080-unit-test-output.sh, so let's move it to t/t0080/t-basic.c and\nadjust Makefiles and .gitignores as necessary.\n\nThis has the additional benefit that test harnesses seeking to run all\nunit tests can find them with a simple glob of \"t/unit-tests/bin/t-*\",\nwith no exceptions needed. This will be important in a later patch where\nwe add support for running the unit tests via a test-tool subcommand.\n\nSigned-off-by: Josh Steadmon \n---\n Makefile | 17 +++++++++++++----\n t/Makefile | 2 +-\n t/t0080-unit-test-output.sh | 24 ++++++++++++------------\n t/t0080/.gitignore | 1 +\n t/{unit-tests => t0080}/t-basic.c | 2 +-\n 5 files changed, 28 insertions(+), 18 deletions(-)\n create mode 100644 t/t0080/.gitignore\n rename t/{unit-tests => t0080}/t-basic.c (98%)\n\ndiff --git a/Makefile b/Makefile\nindex 88ba7a3c51..ab32ec1101 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -683,6 +683,7 @@ TEST_OBJS =\n TEST_PROGRAMS_NEED_X =\n THIRD_PARTY_SOURCES =\n UNIT_TEST_PROGRAMS =\n+UNIT_TEST_HELPERS =\n UNIT_TEST_DIR = t/unit-tests\n UNIT_TEST_BIN = $(UNIT_TEST_DIR)/bin\n \n@@ -1339,10 +1340,12 @@ THIRD_PARTY_SOURCES += compat/regex/%\n THIRD_PARTY_SOURCES += sha1collisiondetection/%\n THIRD_PARTY_SOURCES += sha1dc/%\n \n-UNIT_TEST_PROGRAMS += t-basic\n UNIT_TEST_PROGRAMS += t-strbuf\n UNIT_TEST_PROGS = $(patsubst %,$(UNIT_TEST_BIN)/%$X,$(UNIT_TEST_PROGRAMS))\n+UNIT_TEST_HELPERS += t/t0080/t-basic\n+UNIT_TEST_HELPER_PROGS = $(patsubst %,%$X,$(UNIT_TEST_HELPERS))\n UNIT_TEST_OBJS = $(patsubst %,$(UNIT_TEST_DIR)/%.o,$(UNIT_TEST_PROGRAMS))\n+UNIT_TEST_OBJS += $(patsubst %,%.o,$(UNIT_TEST_HELPERS))\n UNIT_TEST_OBJS += $(UNIT_TEST_DIR)/test-lib.o\n \n # xdiff and reftable libs may in turn depend on what is in libgit.a\n@@ -3189,7 +3192,9 @@ endif\n \n test_bindir_programs := $(patsubst %,bin-wrappers/%,$(BINDIR_PROGRAMS_NEED_X) $(BINDIR_PROGRAMS_NO_X) $(TEST_PROGRAMS_NEED_X))\n \n-all:: $(TEST_PROGRAMS) $(test_bindir_programs) $(UNIT_TEST_PROGS)\n+all:: $(TEST_PROGRAMS) $(test_bindir_programs)\n+\n+all:: $(UNIT_TEST_PROGS) $(UNIT_TEST_HELPER_PROGS)\n \n bin-wrappers/%: wrap-for-bin.sh\n \t$(call mkdir_p_parent_template)\n@@ -3620,7 +3625,7 @@ endif\n \n artifacts-tar:: $(ALL_COMMANDS_TO_INSTALL) $(SCRIPT_LIB) $(OTHER_PROGRAMS) \\\n \t\tGIT-BUILD-OPTIONS $(TEST_PROGRAMS) $(test_bindir_programs) \\\n-\t\t$(UNIT_TEST_PROGS) $(MOFILES)\n+\t\t$(UNIT_TEST_PROGS) $(UNIT_TEST_HELPER_PROGS) $(MOFILES)\n \t$(QUIET_SUBDIR0)templates $(QUIET_SUBDIR1) \\\n \t\tSHELL_PATH='$(SHELL_PATH_SQ)' PERL_PATH='$(PERL_PATH_SQ)'\n \ttest -n \"$(ARTIFACTS_DIRECTORY)\"\n@@ -3682,7 +3687,7 @@ clean: profile-clean coverage-clean cocciclean\n \t$(RM) headless-git.o\n \t$(RM) $(LIB_FILE) $(XDIFF_LIB) $(REFTABLE_LIB) $(REFTABLE_TEST_LIB)\n \t$(RM) $(ALL_PROGRAMS) $(SCRIPT_LIB) $(BUILT_INS) $(OTHER_PROGRAMS)\n-\t$(RM) $(TEST_PROGRAMS) $(UNIT_TEST_PROGS)\n+\t$(RM) $(TEST_PROGRAMS) $(UNIT_TEST_PROGS) $(UNIT_TEST_HELPER_PROGS)\n \t$(RM) $(FUZZ_PROGRAMS)\n \t$(RM) $(SP_OBJ)\n \t$(RM) $(HCC)\n@@ -3869,6 +3874,10 @@ $(UNIT_TEST_PROGS): $(UNIT_TEST_BIN)/%$X: $(UNIT_TEST_DIR)/%.o $(UNIT_TEST_DIR)/\n \t$(QUIET_LINK)$(CC) $(ALL_CFLAGS) -o $@ $(ALL_LDFLAGS) \\\n \t\t$(filter %.o,$^) $(filter %.a,$^) $(LIBS)\n \n+$(UNIT_TEST_HELPER_PROGS): %$X: %.o $(UNIT_TEST_DIR)/test-lib.o $(GITLIBS) GIT-LDFLAGS\n+\t$(QUIET_LINK)$(CC) $(ALL_CFLAGS) -o $@ $(ALL_LDFLAGS) \\\n+\t\t$(filter %.o,$^) $(filter %.a,$^) $(LIBS)\n+\n .PHONY: build-unit-tests unit-tests\n build-unit-tests: $(UNIT_TEST_PROGS)\n unit-tests: $(UNIT_TEST_PROGS)\ndiff --git a/t/Makefile b/t/Makefile\nindex b7a6fefe28..0bee7bc6ea 100644\n--- a/t/Makefile\n+++ b/t/Makefile\n@@ -42,7 +42,7 @@ TPERF = $(sort $(wildcard perf/p[0-9][0-9][0-9][0-9]-*.sh))\n TINTEROP = $(sort $(wildcard interop/i[0-9][0-9][0-9][0-9]-*.sh))\n CHAINLINTTESTS = $(sort $(patsubst chainlint/%.test,%,$(wildcard chainlint/*.test)))\n CHAINLINT = '$(PERL_PATH_SQ)' chainlint.pl\n-UNIT_TESTS = $(sort $(filter-out %.pdb unit-tests/bin/t-basic%,$(wildcard unit-tests/bin/t-*)))\n+UNIT_TESTS = $(sort $(filter-out %.pdb,$(wildcard unit-tests/bin/t-*)))\n \n # `test-chainlint` (which is a dependency of `test-lint`, `test` and `prove`)\n # checks all tests in all scripts via a single invocation, so tell individual\ndiff --git a/t/t0080-unit-test-output.sh b/t/t0080-unit-test-output.sh\nindex 961b54b06c..7431023d97 100755\n--- a/t/t0080-unit-test-output.sh\n+++ b/t/t0080-unit-test-output.sh\n@@ -8,50 +8,50 @@ test_expect_success 'TAP output from unit tests' '\n \tcat >expect <<-EOF &&\n \tok 1 - passing test\n \tok 2 - passing test and assertion return 1\n-\t# check \"1 == 2\" failed at t/unit-tests/t-basic.c:76\n+\t# check \"1 == 2\" failed at t/t0080/t-basic.c:76\n \t# left: 1\n \t# right: 2\n \tnot ok 3 - failing test\n \tok 4 - failing test and assertion return 0\n \tnot ok 5 - passing TEST_TODO() # TODO\n \tok 6 - passing TEST_TODO() returns 1\n-\t# todo check ${SQ}check(x)${SQ} succeeded at t/unit-tests/t-basic.c:25\n+\t# todo check ${SQ}check(x)${SQ} succeeded at t/t0080/t-basic.c:25\n \tnot ok 7 - failing TEST_TODO()\n \tok 8 - failing TEST_TODO() returns 0\n-\t# check \"0\" failed at t/unit-tests/t-basic.c:30\n+\t# check \"0\" failed at t/t0080/t-basic.c:30\n \t# skipping test - missing prerequisite\n-\t# skipping check ${SQ}1${SQ} at t/unit-tests/t-basic.c:32\n+\t# skipping check ${SQ}1${SQ} at t/t0080/t-basic.c:32\n \tok 9 - test_skip() # SKIP\n \tok 10 - skipped test returns 1\n \t# skipping test - missing prerequisite\n \tok 11 - test_skip() inside TEST_TODO() # SKIP\n \tok 12 - test_skip() inside TEST_TODO() returns 1\n-\t# check \"0\" failed at t/unit-tests/t-basic.c:48\n+\t# check \"0\" failed at t/t0080/t-basic.c:48\n \tnot ok 13 - TEST_TODO() after failing check\n \tok 14 - TEST_TODO() after failing check returns 0\n-\t# check \"0\" failed at t/unit-tests/t-basic.c:56\n+\t# check \"0\" failed at t/t0080/t-basic.c:56\n \tnot ok 15 - failing check after TEST_TODO()\n \tok 16 - failing check after TEST_TODO() returns 0\n-\t# check \"!strcmp(\"\\thello\\\\\\\\\", \"there\\\"\\n\")\" failed at t/unit-tests/t-basic.c:61\n+\t# check \"!strcmp(\"\\thello\\\\\\\\\", \"there\\\"\\n\")\" failed at t/t0080/t-basic.c:61\n \t# left: \"\\011hello\\\\\\\\\"\n \t# right: \"there\\\"\\012\"\n-\t# check \"!strcmp(\"NULL\", NULL)\" failed at t/unit-tests/t-basic.c:62\n+\t# check \"!strcmp(\"NULL\", NULL)\" failed at t/t0080/t-basic.c:62\n \t# left: \"NULL\"\n \t# right: NULL\n-\t# check \"${SQ}a${SQ} == ${SQ}\\n${SQ}\" failed at t/unit-tests/t-basic.c:63\n+\t# check \"${SQ}a${SQ} == ${SQ}\\n${SQ}\" failed at t/t0080/t-basic.c:63\n \t# left: ${SQ}a${SQ}\n \t# right: ${SQ}\\012${SQ}\n-\t# check \"${SQ}\\\\\\\\${SQ} == ${SQ}\\\\${SQ}${SQ}\" failed at t/unit-tests/t-basic.c:64\n+\t# check \"${SQ}\\\\\\\\${SQ} == ${SQ}\\\\${SQ}${SQ}\" failed at t/t0080/t-basic.c:64\n \t# left: ${SQ}\\\\\\\\${SQ}\n \t# right: ${SQ}\\\\${SQ}${SQ}\n \tnot ok 17 - messages from failing string and char comparison\n-\t# BUG: test has no checks at t/unit-tests/t-basic.c:91\n+\t# BUG: test has no checks at t/t0080/t-basic.c:91\n \tnot ok 18 - test with no checks\n \tok 19 - test with no checks returns 0\n \t1..19\n \tEOF\n \n-\t! \"$GIT_BUILD_DIR\"/t/unit-tests/bin/t-basic >actual &&\n+\t! \"$GIT_BUILD_DIR\"/t/t0080/t-basic >actual &&\n \ttest_cmp expect actual\n '\n \ndiff --git a/t/t0080/.gitignore b/t/t0080/.gitignore\nnew file mode 100644\nindex 0000000000..1903542827\n--- /dev/null\n+++ b/t/t0080/.gitignore\n@@ -0,0 +1 @@\n+/t-basic\ndiff --git a/t/unit-tests/t-basic.c b/t/t0080/t-basic.c\nsimilarity index 98%\nrename from t/unit-tests/t-basic.c\nrename to t/t0080/t-basic.c\nindex fda1ae59a6..83727221b1 100644\n--- a/t/unit-tests/t-basic.c\n+++ b/t/t0080/t-basic.c\n@@ -1,4 +1,4 @@\n-#include \"test-lib.h\"\n+#include \"../unit-tests/test-lib.h\"\n \n /*\n * The purpose of this \"unit test\" is to verify a few invariants of the unit\n-- \n2.43.0.381.gb435a96ce8-goog\n\n"},{"id":"486892","messageId":"5ecbc976e6216b941e760e096e166ab432ee7784.1705443632.git.steadmon@google.com","threadId":"60754","inReplyTo":"cover.1705443632.git.steadmon@google.com","subject":"[RFC PATCH 2/4] test-tool run-command testsuite: support unit tests","fromName":"Josh Steadmon","fromEmail":"steadmon@google.com","sentAt":"2024-01-16T22:22:59Z","receivedAt":"2024-01-16T22:23:07Z","isPatch":true,"sender":{"key":"steadmon@google.com","avatar":"https://avatars.githubusercontent.com/u/2654920?v=4"},"body":"Teach the testsuite runner in `test-tool run-command testsuite` how to\nrun unit tests, by adding two new flags:\n\nFirst, \"--(no-)run-in-shell\" allows the test-tool to exec the unit-test\nbinaries directly, rather than trying to interpret them as shell\nscripts.\n\nSecond \"--(no-)require-shell-test-pattern\" bypasses the check that the\ntest filenames match the expected t####-*.sh pattern.\n\nWith these changes, you can now use test-tool to run the unit tests:\n$ make\n$ cd t/unit-tests/bin\n$ ../../helper/test-tool run-command testsuite --no-run-in-shell \\\n --no-require-shell-test-pattern\n\nThis should be helpful on Windows to allow running tests without\nrequiring Perl (for `prove`), as discussed in [1] and [2].\n\n[1] https://lore.kernel.org/git/nycvar.QRO.7.76.6.2109091323150.59@tvgsbejvaqbjf.bet/\n[2] https://lore.kernel.org/git/850ea42c-f103-68d5-896b-9120e2628686@gmx.de/\n\nSigned-off-by: Josh Steadmon \n---\n t/helper/test-run-command.c | 40 +++++++++++++++++++++++++++++--------\n 1 file changed, 32 insertions(+), 8 deletions(-)\n\ndiff --git a/t/helper/test-run-command.c b/t/helper/test-run-command.c\nindex c0ed8722c8..2db7e1ef03 100644\n--- a/t/helper/test-run-command.c\n+++ b/t/helper/test-run-command.c\n@@ -64,11 +64,12 @@ static int task_finished(int result UNUSED,\n struct testsuite {\n \tstruct string_list tests, failed;\n \tint next;\n-\tint quiet, immediate, verbose, verbose_log, trace, write_junit_xml;\n+\tint quiet, immediate, verbose, verbose_log, trace, write_junit_xml, run_in_shell;\n };\n #define TESTSUITE_INIT { \\\n \t.tests = STRING_LIST_INIT_DUP, \\\n \t.failed = STRING_LIST_INIT_DUP, \\\n+\t.run_in_shell = 1, \\\n }\n \n static int next_test(struct child_process *cp, struct strbuf *err, void *cb,\n@@ -80,7 +81,9 @@ static int next_test(struct child_process *cp, struct strbuf *err, void *cb,\n \t\treturn 0;\n \n \ttest = suite->tests.items[suite->next++].string;\n-\tstrvec_pushl(&cp->args, \"sh\", test, NULL);\n+\tif (suite->run_in_shell)\n+\t\tstrvec_push(&cp->args, \"sh\");\n+\tstrvec_push(&cp->args, test);\n \tif (suite->quiet)\n \t\tstrvec_push(&cp->args, \"--quiet\");\n \tif (suite->immediate)\n@@ -133,7 +136,7 @@ static const char * const testsuite_usage[] = {\n static int testsuite(int argc, const char **argv)\n {\n \tstruct testsuite suite = TESTSUITE_INIT;\n-\tint max_jobs = 1, i, ret = 0;\n+\tint max_jobs = 1, i, ret = 0, require_shell_test_pattern = 1;\n \tDIR *dir;\n \tstruct dirent *d;\n \tstruct option options[] = {\n@@ -147,6 +150,10 @@ static int testsuite(int argc, const char **argv)\n \t\tOPT_BOOL('x', \"trace\", &suite.trace, \"trace shell commands\"),\n \t\tOPT_BOOL(0, \"write-junit-xml\", &suite.write_junit_xml,\n \t\t\t \"write JUnit-style XML files\"),\n+\t\tOPT_BOOL(0, \"run-in-shell\", &suite.run_in_shell,\n+\t\t\t \"run programs in the suite via `sh`\"),\n+\t\tOPT_BOOL(0, \"require-shell-test-pattern\", &require_shell_test_pattern,\n+\t\t\t \"require programs to match 't####-*.sh'\"),\n \t\tOPT_END()\n \t};\n \tstruct run_process_parallel_opts opts = {\n@@ -155,12 +162,21 @@ static int testsuite(int argc, const char **argv)\n \t\t.task_finished = test_finished,\n \t\t.data = &suite,\n \t};\n+\tstruct strbuf progpath = STRBUF_INIT;\n+\tsize_t path_prefix_len;\n \n \targc = parse_options(argc, argv, NULL, options,\n \t\t\ttestsuite_usage, PARSE_OPT_STOP_AT_NON_OPTION);\n \n \tif (max_jobs <= 0)\n \t\tmax_jobs = online_cpus();\n+\t/*\n+\t * If we run without a shell, we have to provide the relative path to\n+\t * the executables.\n+\t */\n+\tif (!suite.run_in_shell)\n+\t\tstrbuf_addstr(&progpath, \"./\");\n+\tpath_prefix_len = progpath.len;\n \n \tdir = opendir(\".\");\n \tif (!dir)\n@@ -168,20 +184,27 @@ static int testsuite(int argc, const char **argv)\n \twhile ((d = readdir(dir))) {\n \t\tconst char *p = d->d_name;\n \n-\t\tif (*p != 't' || !isdigit(p[1]) || !isdigit(p[2]) ||\n-\t\t !isdigit(p[3]) || !isdigit(p[4]) || p[5] != '-' ||\n-\t\t !ends_with(p, \".sh\"))\n+\t\tif (!strcmp(p, \".\") || !strcmp(p, \"..\"))\n \t\t\tcontinue;\n+\t\tif (require_shell_test_pattern)\n+\t\t\tif (*p != 't' || !isdigit(p[1]) || !isdigit(p[2]) ||\n+\t\t\t !isdigit(p[3]) || !isdigit(p[4]) || p[5] != '-' ||\n+\t\t\t !ends_with(p, \".sh\"))\n+\t\t\t\tcontinue;\n \n \t\t/* No pattern: match all */\n \t\tif (!argc) {\n-\t\t\tstring_list_append(&suite.tests, p);\n+\t\t\tstrbuf_setlen(&progpath, path_prefix_len);\n+\t\t\tstrbuf_addstr(&progpath, p);\n+\t\t\tstring_list_append(&suite.tests, progpath.buf);\n \t\t\tcontinue;\n \t\t}\n \n \t\tfor (i = 0; i < argc; i++)\n \t\t\tif (!wildmatch(argv[i], p, 0)) {\n-\t\t\t\tstring_list_append(&suite.tests, p);\n+\t\t\t\tstrbuf_setlen(&progpath, path_prefix_len);\n+\t\t\t\tstrbuf_addstr(&progpath, p);\n+\t\t\t\tstring_list_append(&suite.tests, progpath.buf);\n \t\t\t\tbreak;\n \t\t\t}\n \t}\n@@ -208,6 +231,7 @@ static int testsuite(int argc, const char **argv)\n \n \tstring_list_clear(&suite.tests, 0);\n \tstring_list_clear(&suite.failed, 0);\n+\tstrbuf_release(&progpath);\n \n \treturn ret;\n }\n-- \n2.43.0.381.gb435a96ce8-goog\n\n"},{"id":"486893","messageId":"5b34c851cdcf97e52a4df0e4bf4724d388c350ce.1705443632.git.steadmon@google.com","threadId":"60754","inReplyTo":"cover.1705443632.git.steadmon@google.com","subject":"[RFC PATCH 3/4] unit tests: add rule for running with test-tool","fromName":"Josh Steadmon","fromEmail":"steadmon@google.com","sentAt":"2024-01-16T22:23:00Z","receivedAt":"2024-01-16T22:23:09Z","isPatch":true,"sender":{"key":"steadmon@google.com","avatar":"https://avatars.githubusercontent.com/u/2654920?v=4"},"body":"In the previous commit, we added support in test-tool for running\ncollections of unit tests. Now, add rules in t/Makefile for running in\nthis way.\n\nThis new rule can be executed from the top-level Makefile via\n`make DEFAULT_UNIT_TEST_TARGET=unit-tests-test-tool unit-tests`, or by\nsetting DEFAULT_UNIT_TEST_TARGET in config.mak.\n\nNEEDS WORK: we need to exclude .pdb files generated by cmake [see\n0df903d402 (unit-tests: do not mistake `.pdb` files for being\nexecutable, 2023-09-25)]\n\nSigned-off-by: Josh Steadmon \n---\n Makefile | 2 +-\n t/Makefile | 9 ++++++++-\n 2 files changed, 9 insertions(+), 2 deletions(-)\n\ndiff --git a/Makefile b/Makefile\nindex ab32ec1101..ce43ad2ae8 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -3880,5 +3880,5 @@ $(UNIT_TEST_HELPER_PROGS): %$X: %.o $(UNIT_TEST_DIR)/test-lib.o $(GITLIBS) GIT-L\n \n .PHONY: build-unit-tests unit-tests\n build-unit-tests: $(UNIT_TEST_PROGS)\n-unit-tests: $(UNIT_TEST_PROGS)\n+unit-tests: $(UNIT_TEST_PROGS) t/helper/test-tool$X\n \t$(MAKE) -C t/ unit-tests\ndiff --git a/t/Makefile b/t/Makefile\nindex 0bee7bc6ea..ad57ec0a41 100644\n--- a/t/Makefile\n+++ b/t/Makefile\n@@ -70,7 +70,7 @@ $(T):\n $(UNIT_TESTS):\n \t@echo \"*** $@ ***\"; $@\n \n-.PHONY: unit-tests unit-tests-raw unit-tests-prove\n+.PHONY: unit-tests unit-tests-raw unit-tests-prove unit-tests-test-tool\n unit-tests: $(DEFAULT_UNIT_TEST_TARGET)\n \n unit-tests-raw: $(UNIT_TESTS)\n@@ -78,6 +78,13 @@ unit-tests-raw: $(UNIT_TESTS)\n unit-tests-prove:\n \t@echo \"*** prove - unit tests ***\"; $(PROVE) $(GIT_PROVE_OPTS) $(UNIT_TESTS)\n \n+unit-tests-test-tool:\n+\t@echo \"*** test-tool - unit tests **\"\n+\t( \\\n+\t\tcd unit-tests/bin && \\\n+\t\t../../helper/test-tool run-command testsuite --no-run-in-shell --no-require-shell-test-pattern \\\n+\t)\n+\n pre-clean:\n \t$(RM) -r '$(TEST_RESULTS_DIRECTORY_SQ)'\n \n-- \n2.43.0.381.gb435a96ce8-goog\n\n"},{"id":"486894","messageId":"c823265f0d6710f99668268ba9ae7092740efbab.1705443632.git.steadmon@google.com","threadId":"60754","inReplyTo":"cover.1705443632.git.steadmon@google.com","subject":"[RFC PATCH 4/4] t/Makefile: run unit tests alongside shell tests","fromName":"Josh Steadmon","fromEmail":"steadmon@google.com","sentAt":"2024-01-16T22:23:01Z","receivedAt":"2024-01-16T22:23:11Z","isPatch":true,"sender":{"key":"steadmon@google.com","avatar":"https://avatars.githubusercontent.com/u/2654920?v=4"},"body":"From: Jeff King \n\nAdd a wrapper script to allow `prove` to run both shell tests and unit\ntests from a single invocation. This avoids issues around running prove\ntwice in CI, as discussed in [1].\n\nAdditionally, this moves the unit tests into the main dev workflow, so\nthat errors can be spotted more quickly.\n\nNEEDS WORK: as discussed in previous commits in this series, there's a\ndesire to avoid `prove` specifically and (IIUC) unnecessary\nfork()/exec()ing in general on Windows. This change adds an extra exec()\nfor each shell and unit test execution, will that be a problem for\nWindows?\n\n[1] https://lore.kernel.org/git/pull.1613.git.1699894837844.gitgitgadget@gmail.com/\n\nSigned-off-by: Jeff King \nSigned-off-by: Josh Steadmon \n---\n t/Makefile | 2 +-\n t/run-test.sh | 13 +++++++++++++\n 2 files changed, 14 insertions(+), 1 deletion(-)\n create mode 100755 t/run-test.sh\n\ndiff --git a/t/Makefile b/t/Makefile\nindex ad57ec0a41..0038a25e33 100644\n--- a/t/Makefile\n+++ b/t/Makefile\n@@ -61,7 +61,7 @@ failed:\n \ttest -z \"$$failed\" || $(MAKE) $$failed\n \n prove: pre-clean check-chainlint $(TEST_LINT)\n-\t@echo \"*** prove ***\"; $(CHAINLINTSUPPRESS) $(PROVE) --exec '$(TEST_SHELL_PATH_SQ)' $(GIT_PROVE_OPTS) $(T) :: $(GIT_TEST_OPTS)\n+\t@echo \"*** prove (shell & unit tests) ***\"; $(CHAINLINTSUPPRESS) $(PROVE) --exec ./run-test.sh $(GIT_PROVE_OPTS) $(T) $(UNIT_TESTS) :: $(GIT_TEST_OPTS)\n \t$(MAKE) clean-except-prove-cache\n \n $(T):\ndiff --git a/t/run-test.sh b/t/run-test.sh\nnew file mode 100755\nindex 0000000000..c29fef48dc\n--- /dev/null\n+++ b/t/run-test.sh\n@@ -0,0 +1,13 @@\n+#!/bin/sh\n+\n+# A simple wrapper to run shell tests via TEST_SHELL_PATH,\n+# or exec unit tests directly.\n+\n+case \"$1\" in\n+*.sh)\n+\texec ${TEST_SHELL_PATH:-/bin/sh} \"$@\"\n+\t;;\n+*)\n+\texec \"$@\"\n+\t;;\n+esac\n-- \n2.43.0.381.gb435a96ce8-goog\n\n"},{"id":"486897","messageId":"xmqqa5p4yjmq.fsf@gitster.g","threadId":"60754","inReplyTo":"a9f67ed703c8314f0f0507ffb83b503717b846b3.1705443632.git.steadmon@google.com","subject":"Re: [RFC PATCH 1/4] t0080: turn t-basic unit test into a helper","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2024-01-16T22:54:21Z","receivedAt":"2024-01-16T22:54:26Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Josh Steadmon writes:\n\n> As such, it makes more sense for t-basic to be a helper item for\n> t0080-unit-test-output.sh, ...\n\nUp to this part was very understandable and agreeable, but I was\nsurprised to see ...\n\n> ... so let's move it to t/t0080/t-basic.c and\n> adjust Makefiles and .gitignores as necessary.\n\n... this conclusion. I somehow thought that t-basic part would be a\ngood test-tool subcommand, as it is run from the suite of shell\nscripts for end-to-end testing CLI interaction.\n\nDo we have any precedent to place programs placed under t/tXXXX/ and\nget them compiled?\n\n> This has the additional benefit that test harnesses seeking to run all\n> unit tests can find them with a simple glob of \"t/unit-tests/bin/t-*\",\n> with no exceptions needed.\n\nAnd this motivation is very much understandable, and for that, as\nlong as it is outside t/unit-tests/, it would be good. I just did\nnot expect it to hang under t/t0080/, inviting more unit test pieces\nthat are triggerable from the numberd shell scripts to hang under\nrandom t/t[0-9][0-9][0-9][0-9]/ directories.\n\n\n"},{"id":"486898","messageId":"xmqqv87sx3y2.fsf@gitster.g","threadId":"60754","inReplyTo":"5ecbc976e6216b941e760e096e166ab432ee7784.1705443632.git.steadmon@google.com","subject":"Re: [RFC PATCH 2/4] test-tool run-command testsuite: support unit tests","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2024-01-16T23:18:29Z","receivedAt":"2024-01-16T23:18:35Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Josh Steadmon writes:\n\n> Teach the testsuite runner in `test-tool run-command testsuite` how to\n> run unit tests, by adding two new flags:\n>\n> First, \"--(no-)run-in-shell\" allows the test-tool to exec the unit-test\n> binaries directly, rather than trying to interpret them as shell\n> scripts.\n\nMakes perfect sense.\n\n> Second \"--(no-)require-shell-test-pattern\" bypasses the check that the\n> test filenames match the expected t####-*.sh pattern.\n\nThis one I am not so sure. Do we still have situations where\nerroring out when fed a non t[0-9][0-9][0-9][0-9]-*.sh script is\nproblematic? IOW, do we need to keep it as conditional?\n\n\t... goes and looks ...\n\nAh, this variable/option is misnamed and that is what invited the\nabove nonsense question out of me. The logic this option disables\ndoes not \"require\" (and error out if the requirement is not met); it\nis used in a loop over \"ls *\" and \"filtering\" files out that are not\nthe numbered scripts.\n\nBut that confusion makes me wonder if non-script side of tests would\nalso want some filtering in the longer run, even if the directory we\nfeed to \"test-tool run\" happens to contain nothing but what we want\nto run right now. I wonder if we instead want a variable that holds\na pattern used to match programs readdir() discovers and skip those\nthat do not match the pattern? Its default value may be something\nlike \"t[0-9][0-9][0-9][0-9]-*.sh\" but alternatively you can give,\nsay, \"*\" to pass everything, or something like that.\n\n> With these changes, you can now use test-tool to run the unit tests:\n> $ make\n> $ cd t/unit-tests/bin\n> $ ../../helper/test-tool run-command testsuite --no-run-in-shell \\\n> --no-require-shell-test-pattern\n\nThis makes me wonder why we want to do the readdir() loop ourselves.\nInstead of saying --no-require-shell-test-pattern there, wouldn't it\nbe simpler to say \"*\" right there, and have testsuite() run the test\nprograms named from the command line?\n\nBut that is orthogonal to the enhancement we have here.\n\n"},{"id":"486899","messageId":"xmqqo7dkx3nv.fsf@gitster.g","threadId":"60754","inReplyTo":"cover.1705443632.git.steadmon@google.com","subject":"Re: [RFC PATCH 0/4] test-tool: add unit test suite runner","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2024-01-16T23:24:36Z","receivedAt":"2024-01-16T23:24:42Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Josh Steadmon writes:\n\n> For various reasons (see discussion at [1]) we would like an alternative\n> to `prove` for running test suites (including the unit tests) on\n> Windows.\n>\n> This series extends the existing `test-tool run-command testsuite` to\n> support running unit tests. In addition, it includes some small\n> cleanups: we move t-basic out of the unit-tests directory, and add a\n> test wrapper script to allow unit tests and the shell test suite to run\n> in a single `prove` process.\n>\n> Some known remaining bits of work:\n> * We need to filter out cmake *.pdb files when running with `test-tool`.\n\n;-)\n\nI should have read this before I commented on that \"--require\nis misnamed---what you really want is a custom filter\" thing.\n\n> * We should investigate switching the Windows CI to use `test-tool`\n> instead of prove.\n\nOK.\n\n> * We should determine whether it is confusing or otherwise harmful to\n> people's workflow to have the unit tests run in parallel with shell\n> tests when using prove as the default test target.\n\nIt is probably confusing especially given that the testsuite() thing\ndoes its own parallel spawning of tests while prove does its own.\nIs there a reason to choose unit-tests-test-tool Makefile target on\nprove-capable platforms (other than for testing the test-tool)?\n\nThanks for an entertaining read.\n"},{"id":"486901","messageId":"xmqqil3sx2y6.fsf@gitster.g","threadId":"60754","inReplyTo":"xmqqv87sx3y2.fsf@gitster.g","subject":"Re: [RFC PATCH 2/4] test-tool run-command testsuite: support unit tests","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2024-01-16T23:40:01Z","receivedAt":"2024-01-16T23:40:08Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Junio C Hamano writes:\n\n> Josh Steadmon writes:\n>\n>> Teach the testsuite runner in `test-tool run-command testsuite` how to\n>> run unit tests, by adding two new flags:\n>>\n>> First, \"--(no-)run-in-shell\" allows the test-tool to exec the unit-test\n>> binaries directly, rather than trying to interpret them as shell\n>> scripts.\n>\n> Makes perfect sense.\n\nThis may be a stupid question, but do we even need the current \"push\n'sh' to the strvec\"? If our executable shell scripts run just fine\nwithout, then this may not have to be conditional.\n"},{"id":"487242","messageId":"20240123004306.GA835964@coredump.intra.peff.net","threadId":"60754","inReplyTo":"xmqqa5p4yjmq.fsf@gitster.g","subject":"Re: [RFC PATCH 1/4] t0080: turn t-basic unit test into a helper","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2024-01-23T00:43:06Z","receivedAt":"2024-01-23T00:43:07Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Tue, Jan 16, 2024 at 02:54:21PM -0800, Junio C Hamano wrote:\n\n> > ... so let's move it to t/t0080/t-basic.c and\n> > adjust Makefiles and .gitignores as necessary.\n> \n> ... this conclusion. I somehow thought that t-basic part would be a\n> good test-tool subcommand, as it is run from the suite of shell\n> scripts for end-to-end testing CLI interaction.\n\nHeh, I was about to ask the same thing.\n\nIn particular...\n\n> Do we have any precedent to place programs placed under t/tXXXX/ and\n> get them compiled?\n\n...no, I don't think we do. And quite often I exclude those directories\nwhen grepping around the code base, because there is often code there\nthat is purely used as a data fixture. E.g., t4256 contains a copy of\nmailinfo.c which it uses as input for some of the tests. That code also\nhappens to have out-of-bounds memory reads which we have since fixed in\nthe real mailinfo.c, but of course \"grep\" finds them both. :)\n\nSo I would prefer a rule like \"no buildable code in t/t[0-9]*\". Barring\nthat, maybe we could avoid using things that look too much like real Git\ncode in our tests (though we sometimes do need fake code for things like\nfunclines, and even that might end up creating false positives).\n\n-Peff\n"},{"id":"487244","messageId":"20240123005913.GB835964@coredump.intra.peff.net","threadId":"60754","inReplyTo":"xmqqil3sx2y6.fsf@gitster.g","subject":"Re: [RFC PATCH 2/4] test-tool run-command testsuite: support unit tests","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2024-01-23T00:59:13Z","receivedAt":"2024-01-23T00:59:14Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Tue, Jan 16, 2024 at 03:40:01PM -0800, Junio C Hamano wrote:\n\n> Junio C Hamano writes:\n> \n> > Josh Steadmon writes:\n> >\n> >> Teach the testsuite runner in `test-tool run-command testsuite` how to\n> >> run unit tests, by adding two new flags:\n> >>\n> >> First, \"--(no-)run-in-shell\" allows the test-tool to exec the unit-test\n> >> binaries directly, rather than trying to interpret them as shell\n> >> scripts.\n> >\n> > Makes perfect sense.\n> \n> This may be a stupid question, but do we even need the current \"push\n> 'sh' to the strvec\"? If our executable shell scripts run just fine\n> without, then this may not have to be conditional.\n\nIt is necessary for the same reason that the indirection provided by\npatch 4 is necessary: the test scripts are supposed to be run by\nTEST_SHELL_PATH, even if they may say \"#!/bin/sh\" in their header.\n\nBut the \"testsuite\" helper just hard-codes \"sh\", which is wrong. It's\nsomewhat academic for Windows where \"sh\" is already bash, and that's the\nonly reasonable thing anybody would set TEST_SHELL_PATH to anyway. But\nif the point is to be a drop-in replacement for the existing flow, it\nshould be checking getenv(\"TEST_SHELL_PATH\"). And perhaps the Makefile\nmight need to correctly export the variable.\n\n-Peff\n"},{"id":"487772","messageId":"ZbvzyqpvVlVhVi2Y@google.com","threadId":"60754","inReplyTo":"20240123004306.GA835964@coredump.intra.peff.net","subject":"Re: [RFC PATCH 1/4] t0080: turn t-basic unit test into a helper","fromName":"Josh Steadmon","fromEmail":"steadmon@google.com","sentAt":"2024-02-01T19:40:58Z","receivedAt":"2024-02-01T19:41:03Z","isPatch":true,"sender":{"key":"steadmon@google.com","avatar":"https://avatars.githubusercontent.com/u/2654920?v=4"},"body":"On 2024.01.22 19:43, Jeff King wrote:\n> On Tue, Jan 16, 2024 at 02:54:21PM -0800, Junio C Hamano wrote:\n> \n> > > ... so let's move it to t/t0080/t-basic.c and\n> > > adjust Makefiles and .gitignores as necessary.\n> > \n> > ... this conclusion. I somehow thought that t-basic part would be a\n> > good test-tool subcommand, as it is run from the suite of shell\n> > scripts for end-to-end testing CLI interaction.\n> \n> Heh, I was about to ask the same thing.\n> \n> In particular...\n> \n> > Do we have any precedent to place programs placed under t/tXXXX/ and\n> > get them compiled?\n> \n> ...no, I don't think we do. And quite often I exclude those directories\n> when grepping around the code base, because there is often code there\n> that is purely used as a data fixture. E.g., t4256 contains a copy of\n> mailinfo.c which it uses as input for some of the tests. That code also\n> happens to have out-of-bounds memory reads which we have since fixed in\n> the real mailinfo.c, but of course \"grep\" finds them both. :)\n> \n> So I would prefer a rule like \"no buildable code in t/t[0-9]*\". Barring\n> that, maybe we could avoid using things that look too much like real Git\n> code in our tests (though we sometimes do need fake code for things like\n> funclines, and even that might end up creating false positives).\n> \n> -Peff\n\nAck. I've moved this to a test-tool subcommand for V2, which I hope to\nsend out soon.\n"},{"id":"487778","messageId":"ZbwV9iGsTWZ3ddfn@google.com","threadId":"60754","inReplyTo":"xmqqv87sx3y2.fsf@gitster.g","subject":"Re: [RFC PATCH 2/4] test-tool run-command testsuite: support unit tests","fromName":"Josh Steadmon","fromEmail":"steadmon@google.com","sentAt":"2024-02-01T22:06:46Z","receivedAt":"2024-02-01T22:06:52Z","isPatch":true,"sender":{"key":"steadmon@google.com","avatar":"https://avatars.githubusercontent.com/u/2654920?v=4"},"body":"On 2024.01.16 15:18, Junio C Hamano wrote:\n> Josh Steadmon writes:\n> > With these changes, you can now use test-tool to run the unit tests:\n> > $ make\n> > $ cd t/unit-tests/bin\n> > $ ../../helper/test-tool run-command testsuite --no-run-in-shell \\\n> > --no-require-shell-test-pattern\n> \n> This makes me wonder why we want to do the readdir() loop ourselves.\n> Instead of saying --no-require-shell-test-pattern there, wouldn't it\n> be simpler to say \"*\" right there, and have testsuite() run the test\n> programs named from the command line?\n\nIt's speculation on my part, but I wonder if it has something to do with\nthe number of shell tests? Google tells me that on Windows, the maximum\ncommand line length is 8191 characters. Which is actually a fair bit\nsmaller than expanding the shell test list:\n\n$ echo t????-*.sh | wc -c\n25714\n"},{"id":"487781","messageId":"xmqqle83x1lq.fsf@gitster.g","threadId":"60754","inReplyTo":"ZbwV9iGsTWZ3ddfn@google.com","subject":"Re: [RFC PATCH 2/4] test-tool run-command testsuite: support unit tests","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2024-02-01T22:26:41Z","receivedAt":"2024-02-01T22:26:44Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Josh Steadmon writes:\n\n> On 2024.01.16 15:18, Junio C Hamano wrote:\n>> Josh Steadmon writes:\n>> > With these changes, you can now use test-tool to run the unit tests:\n>> > $ make\n>> > $ cd t/unit-tests/bin\n>> > $ ../../helper/test-tool run-command testsuite --no-run-in-shell \\\n>> > --no-require-shell-test-pattern\n>> \n>> This makes me wonder why we want to do the readdir() loop ourselves.\n>> Instead of saying --no-require-shell-test-pattern there, wouldn't it\n>> be simpler to say \"*\" right there, and have testsuite() run the test\n>> programs named from the command line?\n>\n> It's speculation on my part, but I wonder if it has something to do with\n> the number of shell tests? Google tells me that on Windows, the maximum\n> command line length is 8191 characters. Which is actually a fair bit\n> smaller than expanding the shell test list:\n>\n> $ echo t????-*.sh | wc -c\n> 25714\n\nOK.\n"},{"id":"487782","messageId":"Zbwki_Dk2hGRce6Y@google.com","threadId":"60754","inReplyTo":"xmqqv87sx3y2.fsf@gitster.g","subject":"Re: [RFC PATCH 2/4] test-tool run-command testsuite: support unit tests","fromName":"Josh Steadmon","fromEmail":"steadmon@google.com","sentAt":"2024-02-01T23:08:59Z","receivedAt":"2024-02-01T23:09:05Z","isPatch":true,"sender":{"key":"steadmon@google.com","avatar":"https://avatars.githubusercontent.com/u/2654920?v=4"},"body":"On 2024.01.16 15:18, Junio C Hamano wrote:\n> Josh Steadmon writes:\n> > Second \"--(no-)require-shell-test-pattern\" bypasses the check that the\n> > test filenames match the expected t####-*.sh pattern.\n> \n> This one I am not so sure. Do we still have situations where\n> erroring out when fed a non t[0-9][0-9][0-9][0-9]-*.sh script is\n> problematic? IOW, do we need to keep it as conditional?\n> \n> \t... goes and looks ...\n> \n> Ah, this variable/option is misnamed and that is what invited the\n> above nonsense question out of me. The logic this option disables\n> does not \"require\" (and error out if the requirement is not met); it\n> is used in a loop over \"ls *\" and \"filtering\" files out that are not\n> the numbered scripts.\n> \n> But that confusion makes me wonder if non-script side of tests would\n> also want some filtering in the longer run, even if the directory we\n> feed to \"test-tool run\" happens to contain nothing but what we want\n> to run right now. I wonder if we instead want a variable that holds\n> a pattern used to match programs readdir() discovers and skip those\n> that do not match the pattern? Its default value may be something\n> like \"t[0-9][0-9][0-9][0-9]-*.sh\" but alternatively you can give,\n> say, \"*\" to pass everything, or something like that.\n\nThe original implementation and this series both still allow passing\nadditional patterns on the command line. Only tests matching the\nhardcoded t[0-9][0-9][0-9][0-9]-*.sh pattern plus all the command-line\npatterns will be run. The new flag just bypasses the hardcoded pattern.\n\nGiven that V2 will already be changing in non-backwards-compatible ways,\nI think it makes the most sense to just delete the hardcoded pattern and\nrequire callers to provide it if they want it.\n"},{"id":"487783","messageId":"ZbwlzPWVtYuXy8rY@google.com","threadId":"60754","inReplyTo":"Zbwki_Dk2hGRce6Y@google.com","subject":"Re: [RFC PATCH 2/4] test-tool run-command testsuite: support unit tests","fromName":"Josh Steadmon","fromEmail":"steadmon@google.com","sentAt":"2024-02-01T23:14:20Z","receivedAt":"2024-02-01T23:14:26Z","isPatch":true,"sender":{"key":"steadmon@google.com","avatar":"https://avatars.githubusercontent.com/u/2654920?v=4"},"body":"On 2024.02.01 15:08, Josh Steadmon wrote:\n> On 2024.01.16 15:18, Junio C Hamano wrote:\n> > Josh Steadmon writes:\n> > > Second \"--(no-)require-shell-test-pattern\" bypasses the check that the\n> > > test filenames match the expected t####-*.sh pattern.\n> > \n> > This one I am not so sure. Do we still have situations where\n> > erroring out when fed a non t[0-9][0-9][0-9][0-9]-*.sh script is\n> > problematic? IOW, do we need to keep it as conditional?\n> > \n> > \t... goes and looks ...\n> > \n> > Ah, this variable/option is misnamed and that is what invited the\n> > above nonsense question out of me. The logic this option disables\n> > does not \"require\" (and error out if the requirement is not met); it\n> > is used in a loop over \"ls *\" and \"filtering\" files out that are not\n> > the numbered scripts.\n> > \n> > But that confusion makes me wonder if non-script side of tests would\n> > also want some filtering in the longer run, even if the directory we\n> > feed to \"test-tool run\" happens to contain nothing but what we want\n> > to run right now. I wonder if we instead want a variable that holds\n> > a pattern used to match programs readdir() discovers and skip those\n> > that do not match the pattern? Its default value may be something\n> > like \"t[0-9][0-9][0-9][0-9]-*.sh\" but alternatively you can give,\n> > say, \"*\" to pass everything, or something like that.\n> \n> The original implementation and this series both still allow passing\n> additional patterns on the command line. Only tests matching the\n> hardcoded t[0-9][0-9][0-9][0-9]-*.sh pattern plus all the command-line\n> patterns will be run. The new flag just bypasses the hardcoded pattern.\n\nActually, I misspoke here. The test must match the hardcoded pattern\nplus at least one of the command-line patterns (if any were provided).\n\n> Given that V2 will already be changing in non-backwards-compatible ways,\n> I think it makes the most sense to just delete the hardcoded pattern and\n> require callers to provide it if they want it.\n\nGiven that it sounds like the testsuite functionality is not really used\nat the moment, I still feel OK with changing the pattern matching logic\nhere for V2.\n"},{"id":"487852","messageId":"cover.1706921262.git.steadmon@google.com","threadId":"60754","inReplyTo":"cover.1705443632.git.steadmon@google.com","subject":"[RFC PATCH v2 0/6] test-tool: add unit test suite runner","fromName":"Josh Steadmon","fromEmail":"steadmon@google.com","sentAt":"2024-02-03T00:50:25Z","receivedAt":"2024-02-03T00:50:34Z","isPatch":true,"sender":{"key":"steadmon@google.com","avatar":"https://avatars.githubusercontent.com/u/2654920?v=4"},"body":"Please note: this series has been rebased onto jk/unit-tests-buildfix.\n\nFor various reasons (see discussion at [1]) we would like an alternative\nto `prove` for running test suites (including the unit tests) on\nWindows.\n\nThis series extends the existing `test-tool run-command testsuite` to\nsupport running unit tests. In addition, it includes some small\ncleanups:\n* move t-basic out of the unit-tests directory\n* don't hardcode the shell for running tests in `test-tool ... testsuite`\n* don't hardcode a test name filter in `test-tool ... testsuite`\n* add a test wrapper script to allow unit tests and the shell test suite\n to run in a single `prove` process\n\nSome known remaining bits of work:\n* We should investigate switching the Windows CI to use `test-tool`\n instead of prove. However, Windows CI seems broken on\n jk/unit-tests-buildfix, and I haven't had time to determine why.\n* We should determine whether it is confusing or otherwise harmful to\n people's workflow to have the unit tests run in parallel with shell\n tests when using prove as the default test target.\n\n[1] https://lore.kernel.org/git/pull.1613.git.1699894837844.gitgitgadget@gmail.com/\n\nChanges in V2:\n* Patch 1: move t-basic to a test-tool subcommand rather than a new\n executable under t/t0080/\n* New patch 2: get the shell path from TEST_SHELL_PATH in\n `test-tool run-command testsuite`\n* New patch 3: remove the hardcoded filename filter in\n `test-tool run-command testsuite`\n* Patch 4 (previously 2): simplified now that we no longer need to add\n any command-line flags to support unit tests\n* Patch 5 (previously 3): avoid trying to run cmake *.pdb files by using\n the unit test list built in the makefile in jk/unit-tests-buildfix.\n\n\nJeff King (1):\n t/Makefile: run unit tests alongside shell tests\n\nJosh Steadmon (5):\n t0080: turn t-basic unit test into a helper\n test-tool run-command testsuite: get shell from env\n test-tool run-command testsuite: remove hardcoded filter\n test-tool run-command testsuite: support unit tests\n unit tests: add rule for running with test-tool\n\n Makefile | 6 ++--\n t/Makefile | 15 +++++++---\n .../t-basic.c => helper/test-example-tap.c} | 5 ++--\n t/helper/test-run-command.c | 29 +++++++++++++++----\n t/helper/test-tool.c | 1 +\n t/helper/test-tool.h | 1 +\n t/run-test.sh | 13 +++++++++\n t/t0080-unit-test-output.sh | 24 +++++++--------\n 8 files changed, 67 insertions(+), 27 deletions(-)\n rename t/{unit-tests/t-basic.c => helper/test-example-tap.c} (95%)\n create mode 100755 t/run-test.sh\n\nRange-diff against v1:\n1: a9f67ed703 < -: ---------- t0080: turn t-basic unit test into a helper\n-: ---------- > 1: da756b4bfb t0080: turn t-basic unit test into a helper\n-: ---------- > 2: c8448406d7 test-tool run-command testsuite: get shell from env\n-: ---------- > 3: e1b89ae93e test-tool run-command testsuite: remove hardcoded filter\n2: 5ecbc976e6 ! 4: b5665386b5 test-tool run-command testsuite: support unit tests\n @@ Commit message\n test-tool run-command testsuite: support unit tests\n \n Teach the testsuite runner in `test-tool run-command testsuite` how to\n - run unit tests, by adding two new flags:\n + run unit tests: if TEST_SHELL_PATH is not set, assume that we're running\n + the programs directly from CWD, rather than defaulting to \"sh\" as an\n + interpreter.\n \n - First, \"--(no-)run-in-shell\" allows the test-tool to exec the unit-test\n - binaries directly, rather than trying to interpret them as shell\n - scripts.\n -\n - Second \"--(no-)require-shell-test-pattern\" bypasses the check that the\n - test filenames match the expected t####-*.sh pattern.\n -\n - With these changes, you can now use test-tool to run the unit tests:\n + With this change, you can now use test-tool to run the unit tests:\n $ make\n $ cd t/unit-tests/bin\n - $ ../../helper/test-tool run-command testsuite --no-run-in-shell \\\n - --no-require-shell-test-pattern\n + $ ../../helper/test-tool run-command testsuite\n \n This should be helpful on Windows to allow running tests without\n requiring Perl (for `prove`), as discussed in [1] and [2].\n \n + This again breaks backwards compatibility, as it is now required to set\n + TEST_SHELL_PATH properly for executing shell scripts, but again, as\n + noted in [2], there are no longer any such invocations in our codebase.\n +\n [1] https://lore.kernel.org/git/nycvar.QRO.7.76.6.2109091323150.59@tvgsbejvaqbjf.bet/\n [2] https://lore.kernel.org/git/850ea42c-f103-68d5-896b-9120e2628686@gmx.de/\n \n \n ## t/helper/test-run-command.c ##\n -@@ t/helper/test-run-command.c: static int task_finished(int result UNUSED,\n - struct testsuite {\n - \tstruct string_list tests, failed;\n - \tint next;\n --\tint quiet, immediate, verbose, verbose_log, trace, write_junit_xml;\n -+\tint quiet, immediate, verbose, verbose_log, trace, write_junit_xml, run_in_shell;\n - };\n - #define TESTSUITE_INIT { \\\n - \t.tests = STRING_LIST_INIT_DUP, \\\n - \t.failed = STRING_LIST_INIT_DUP, \\\n -+\t.run_in_shell = 1, \\\n - }\n - \n - static int next_test(struct child_process *cp, struct strbuf *err, void *cb,\n -@@ t/helper/test-run-command.c: static int next_test(struct child_process *cp, struct strbuf *err, void *cb,\n - \t\treturn 0;\n - \n - \ttest = suite->tests.items[suite->next++].string;\n --\tstrvec_pushl(&cp->args, \"sh\", test, NULL);\n -+\tif (suite->run_in_shell)\n -+\t\tstrvec_push(&cp->args, \"sh\");\n -+\tstrvec_push(&cp->args, test);\n - \tif (suite->quiet)\n - \t\tstrvec_push(&cp->args, \"--quiet\");\n - \tif (suite->immediate)\n -@@ t/helper/test-run-command.c: static const char * const testsuite_usage[] = {\n - static int testsuite(int argc, const char **argv)\n - {\n - \tstruct testsuite suite = TESTSUITE_INIT;\n --\tint max_jobs = 1, i, ret = 0;\n -+\tint max_jobs = 1, i, ret = 0, require_shell_test_pattern = 1;\n - \tDIR *dir;\n - \tstruct dirent *d;\n - \tstruct option options[] = {\n -@@ t/helper/test-run-command.c: static int testsuite(int argc, const char **argv)\n - \t\tOPT_BOOL('x', \"trace\", &suite.trace, \"trace shell commands\"),\n - \t\tOPT_BOOL(0, \"write-junit-xml\", &suite.write_junit_xml,\n - \t\t\t \"write JUnit-style XML files\"),\n -+\t\tOPT_BOOL(0, \"run-in-shell\", &suite.run_in_shell,\n -+\t\t\t \"run programs in the suite via `sh`\"),\n -+\t\tOPT_BOOL(0, \"require-shell-test-pattern\", &require_shell_test_pattern,\n -+\t\t\t \"require programs to match 't####-*.sh'\"),\n - \t\tOPT_END()\n - \t};\n - \tstruct run_process_parallel_opts opts = {\n @@ t/helper/test-run-command.c: static int testsuite(int argc, const char **argv)\n \t\t.task_finished = test_finished,\n \t\t.data = &suite,\n @@ t/helper/test-run-command.c: static int testsuite(int argc, const char **argv)\n \n \targc = parse_options(argc, argv, NULL, options,\n \t\t\ttestsuite_usage, PARSE_OPT_STOP_AT_NON_OPTION);\n - \n +@@ t/helper/test-run-command.c: static int testsuite(int argc, const char **argv)\n \tif (max_jobs <= 0)\n \t\tmax_jobs = online_cpus();\n + \n +\t/*\n +\t * If we run without a shell, we have to provide the relative path to\n +\t * the executables.\n +\t */\n -+\tif (!suite.run_in_shell)\n + \tsuite.shell_path = getenv(\"TEST_SHELL_PATH\");\n + \tif (!suite.shell_path)\n +-\t\tsuite.shell_path = \"sh\";\n +\t\tstrbuf_addstr(&progpath, \"./\");\n +\tpath_prefix_len = progpath.len;\n \n \tdir = opendir(\".\");\n \tif (!dir)\n @@ t/helper/test-run-command.c: static int testsuite(int argc, const char **argv)\n - \twhile ((d = readdir(dir))) {\n - \t\tconst char *p = d->d_name;\n - \n --\t\tif (*p != 't' || !isdigit(p[1]) || !isdigit(p[2]) ||\n --\t\t !isdigit(p[3]) || !isdigit(p[4]) || p[5] != '-' ||\n --\t\t !ends_with(p, \".sh\"))\n -+\t\tif (!strcmp(p, \".\") || !strcmp(p, \"..\"))\n - \t\t\tcontinue;\n -+\t\tif (require_shell_test_pattern)\n -+\t\t\tif (*p != 't' || !isdigit(p[1]) || !isdigit(p[2]) ||\n -+\t\t\t !isdigit(p[3]) || !isdigit(p[4]) || p[5] != '-' ||\n -+\t\t\t !ends_with(p, \".sh\"))\n -+\t\t\t\tcontinue;\n \n \t\t/* No pattern: match all */\n \t\tif (!argc) {\n3: 5b34c851cd ! 5: f2746703d5 unit tests: add rule for running with test-tool\n @@ Commit message\n `make DEFAULT_UNIT_TEST_TARGET=unit-tests-test-tool unit-tests`, or by\n setting DEFAULT_UNIT_TEST_TARGET in config.mak.\n \n - NEEDS WORK: we need to exclude .pdb files generated by cmake [see\n - 0df903d402 (unit-tests: do not mistake `.pdb` files for being\n - executable, 2023-09-25)]\n -\n \n ## Makefile ##\n -@@ Makefile: $(UNIT_TEST_HELPER_PROGS): %$X: %.o $(UNIT_TEST_DIR)/test-lib.o $(GITLIBS) GIT-L\n +@@ Makefile: $(UNIT_TEST_PROGS): $(UNIT_TEST_BIN)/%$X: $(UNIT_TEST_DIR)/%.o $(UNIT_TEST_DIR)/\n \n .PHONY: build-unit-tests unit-tests\n build-unit-tests: $(UNIT_TEST_PROGS)\n @@ Makefile: $(UNIT_TEST_HELPER_PROGS): %$X: %.o $(UNIT_TEST_DIR)/test-lib.o $(GITL\n \t$(MAKE) -C t/ unit-tests\n \n ## t/Makefile ##\n +@@ t/Makefile: CHAINLINTTESTS = $(sort $(patsubst chainlint/%.test,%,$(wildcard chainlint/*.tes\n + CHAINLINT = '$(PERL_PATH_SQ)' chainlint.pl\n + UNIT_TEST_SOURCES = $(wildcard unit-tests/t-*.c)\n + UNIT_TESTS = $(patsubst unit-tests/%.c,unit-tests/bin/%$(X),$(UNIT_TEST_SOURCES))\n ++UNIT_TESTS_NO_DIR = $(notdir $(UNIT_TESTS))\n + \n + # `test-chainlint` (which is a dependency of `test-lint`, `test` and `prove`)\n + # checks all tests in all scripts via a single invocation, so tell individual\n @@ t/Makefile: $(T):\n $(UNIT_TESTS):\n \t@echo \"*** $@ ***\"; $@\n @@ t/Makefile: unit-tests-raw: $(UNIT_TESTS)\n +\t@echo \"*** test-tool - unit tests **\"\n +\t( \\\n +\t\tcd unit-tests/bin && \\\n -+\t\t../../helper/test-tool run-command testsuite --no-run-in-shell --no-require-shell-test-pattern \\\n ++\t\t../../helper/test-tool$X run-command testsuite $(UNIT_TESTS_NO_DIR)\\\n +\t)\n +\n pre-clean:\n4: c823265f0d = 6: cd7467a7bd t/Makefile: run unit tests alongside shell tests\n\nbase-commit: 799d449105dc1f6e77fa1ebaea4f6d8bdc6537cf\n-- \n2.43.0.594.gd9cf4e227d-goog\n\n"},{"id":"487853","messageId":"da756b4bfb9d1ce0d1213d585e72acfbf667e2a2.1706921262.git.steadmon@google.com","threadId":"60754","inReplyTo":"cover.1706921262.git.steadmon@google.com","subject":"[RFC PATCH v2 1/6] t0080: turn t-basic unit test into a helper","fromName":"Josh Steadmon","fromEmail":"steadmon@google.com","sentAt":"2024-02-03T00:50:26Z","receivedAt":"2024-02-03T00:50:37Z","isPatch":true,"sender":{"key":"steadmon@google.com","avatar":"https://avatars.githubusercontent.com/u/2654920?v=4"},"body":"While t/unit-tests/t-basic.c uses the unit-test framework added in\ne137fe3b29 (unit tests: add TAP unit test framework, 2023-11-09), it is\nnot a true unit test in that it intentionally fails in order to exercise\nvarious codepaths in the unit-test framework. Thus, we intentionally\nexclude it when running unit tests through the various t/Makefile\ntargets. Instead, it is executed by t0080-unit-test-output.sh, which\nverifies its output follows the TAP format expected for the various\npass, skip, or fail cases.\n\nAs such, it makes more sense for t-basic to be a helper item for\nt0080-unit-test-output.sh, so let's move it to\nt/helper/test-example-tap.c and adjust Makefiles as necessary.\n\nThis has the additional benefit that test harnesses seeking to run all\nunit tests can find them with a simple glob of \"t/unit-tests/bin/t-*\",\nwith no exceptions needed. This will be important in a later patch where\nwe add support for running the unit tests via a test-tool subcommand.\n\nSigned-off-by: Josh Steadmon \n---\n Makefile | 4 ++--\n t/Makefile | 3 +--\n .../t-basic.c => helper/test-example-tap.c} | 5 ++--\n t/helper/test-tool.c | 1 +\n t/helper/test-tool.h | 1 +\n t/t0080-unit-test-output.sh | 24 +++++++++----------\n 6 files changed, 20 insertions(+), 18 deletions(-)\n rename t/{unit-tests/t-basic.c => helper/test-example-tap.c} (95%)\n\ndiff --git a/Makefile b/Makefile\nindex 23723367b8..ba55d817ee 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -802,6 +802,7 @@ TEST_BUILTINS_OBJS += test-dump-split-index.o\n TEST_BUILTINS_OBJS += test-dump-untracked-cache.o\n TEST_BUILTINS_OBJS += test-env-helper.o\n TEST_BUILTINS_OBJS += test-example-decorate.o\n+TEST_BUILTINS_OBJS += test-example-tap.o\n TEST_BUILTINS_OBJS += test-find-pack.o\n TEST_BUILTINS_OBJS += test-fsmonitor-client.o\n TEST_BUILTINS_OBJS += test-genrandom.o\n@@ -1338,7 +1339,6 @@ THIRD_PARTY_SOURCES += compat/regex/%\n THIRD_PARTY_SOURCES += sha1collisiondetection/%\n THIRD_PARTY_SOURCES += sha1dc/%\n \n-UNIT_TEST_PROGRAMS += t-basic\n UNIT_TEST_PROGRAMS += t-mem-pool\n UNIT_TEST_PROGRAMS += t-strbuf\n UNIT_TEST_PROGRAMS += t-ctype\n@@ -3218,7 +3218,7 @@ perf: all\n \n .PRECIOUS: $(TEST_OBJS)\n \n-t/helper/test-tool$X: $(patsubst %,t/helper/%,$(TEST_BUILTINS_OBJS))\n+t/helper/test-tool$X: $(patsubst %,t/helper/%,$(TEST_BUILTINS_OBJS)) $(UNIT_TEST_DIR)/test-lib.o\n \n t/helper/test-%$X: t/helper/test-%.o GIT-LDFLAGS $(GITLIBS) $(REFTABLE_TEST_LIB)\n \t$(QUIET_LINK)$(CC) $(ALL_CFLAGS) -o $@ $(ALL_LDFLAGS) $(filter %.o,$^) $(filter %.a,$^) $(LIBS)\ndiff --git a/t/Makefile b/t/Makefile\nindex 281f4c3534..1283c90c10 100644\n--- a/t/Makefile\n+++ b/t/Makefile\n@@ -44,8 +44,7 @@ TINTEROP = $(sort $(wildcard interop/i[0-9][0-9][0-9][0-9]-*.sh))\n CHAINLINTTESTS = $(sort $(patsubst chainlint/%.test,%,$(wildcard chainlint/*.test)))\n CHAINLINT = '$(PERL_PATH_SQ)' chainlint.pl\n UNIT_TEST_SOURCES = $(wildcard unit-tests/t-*.c)\n-UNIT_TEST_PROGRAMS = $(patsubst unit-tests/%.c,unit-tests/bin/%$(X),$(UNIT_TEST_SOURCES))\n-UNIT_TESTS = $(sort $(filter-out unit-tests/bin/t-basic%,$(UNIT_TEST_PROGRAMS)))\n+UNIT_TESTS = $(patsubst unit-tests/%.c,unit-tests/bin/%$(X),$(UNIT_TEST_SOURCES))\n \n # `test-chainlint` (which is a dependency of `test-lint`, `test` and `prove`)\n # checks all tests in all scripts via a single invocation, so tell individual\ndiff --git a/t/unit-tests/t-basic.c b/t/helper/test-example-tap.c\nsimilarity index 95%\nrename from t/unit-tests/t-basic.c\nrename to t/helper/test-example-tap.c\nindex fda1ae59a6..21e4848e78 100644\n--- a/t/unit-tests/t-basic.c\n+++ b/t/helper/test-example-tap.c\n@@ -1,4 +1,5 @@\n-#include \"test-lib.h\"\n+#include \"t/unit-tests/test-lib.h\"\n+#include \"test-tool.h\"\n \n /*\n * The purpose of this \"unit test\" is to verify a few invariants of the unit\n@@ -69,7 +70,7 @@ static void t_empty(void)\n \t; /* empty */\n }\n \n-int cmd_main(int argc, const char **argv)\n+int cmd__example_tap(int argc, const char **argv)\n {\n \ttest_res = TEST(check_res = check_int(1, ==, 1), \"passing test\");\n \tTEST(t_res(1), \"passing test and assertion return 1\");\ndiff --git a/t/helper/test-tool.c b/t/helper/test-tool.c\nindex 33b9501c21..bb5c04c9c0 100644\n--- a/t/helper/test-tool.c\n+++ b/t/helper/test-tool.c\n@@ -29,6 +29,7 @@ static struct test_cmd cmds[] = {\n \t{ \"dump-untracked-cache\", cmd__dump_untracked_cache },\n \t{ \"env-helper\", cmd__env_helper },\n \t{ \"example-decorate\", cmd__example_decorate },\n+\t{ \"example-tap\", cmd__example_tap },\n \t{ \"find-pack\", cmd__find_pack },\n \t{ \"fsmonitor-client\", cmd__fsmonitor_client },\n \t{ \"genrandom\", cmd__genrandom },\ndiff --git a/t/helper/test-tool.h b/t/helper/test-tool.h\nindex b72f07ded9..38001bd1c6 100644\n--- a/t/helper/test-tool.h\n+++ b/t/helper/test-tool.h\n@@ -23,6 +23,7 @@ int cmd__dump_untracked_cache(int argc, const char **argv);\n int cmd__dump_reftable(int argc, const char **argv);\n int cmd__env_helper(int argc, const char **argv);\n int cmd__example_decorate(int argc, const char **argv);\n+int cmd__example_tap(int argc, const char **argv);\n int cmd__find_pack(int argc, const char **argv);\n int cmd__fsmonitor_client(int argc, const char **argv);\n int cmd__genrandom(int argc, const char **argv);\ndiff --git a/t/t0080-unit-test-output.sh b/t/t0080-unit-test-output.sh\nindex 961b54b06c..83b1e3b7f5 100755\n--- a/t/t0080-unit-test-output.sh\n+++ b/t/t0080-unit-test-output.sh\n@@ -8,50 +8,50 @@ test_expect_success 'TAP output from unit tests' '\n \tcat >expect <<-EOF &&\n \tok 1 - passing test\n \tok 2 - passing test and assertion return 1\n-\t# check \"1 == 2\" failed at t/unit-tests/t-basic.c:76\n+\t# check \"1 == 2\" failed at t/helper/test-example-tap.c:77\n \t# left: 1\n \t# right: 2\n \tnot ok 3 - failing test\n \tok 4 - failing test and assertion return 0\n \tnot ok 5 - passing TEST_TODO() # TODO\n \tok 6 - passing TEST_TODO() returns 1\n-\t# todo check ${SQ}check(x)${SQ} succeeded at t/unit-tests/t-basic.c:25\n+\t# todo check ${SQ}check(x)${SQ} succeeded at t/helper/test-example-tap.c:26\n \tnot ok 7 - failing TEST_TODO()\n \tok 8 - failing TEST_TODO() returns 0\n-\t# check \"0\" failed at t/unit-tests/t-basic.c:30\n+\t# check \"0\" failed at t/helper/test-example-tap.c:31\n \t# skipping test - missing prerequisite\n-\t# skipping check ${SQ}1${SQ} at t/unit-tests/t-basic.c:32\n+\t# skipping check ${SQ}1${SQ} at t/helper/test-example-tap.c:33\n \tok 9 - test_skip() # SKIP\n \tok 10 - skipped test returns 1\n \t# skipping test - missing prerequisite\n \tok 11 - test_skip() inside TEST_TODO() # SKIP\n \tok 12 - test_skip() inside TEST_TODO() returns 1\n-\t# check \"0\" failed at t/unit-tests/t-basic.c:48\n+\t# check \"0\" failed at t/helper/test-example-tap.c:49\n \tnot ok 13 - TEST_TODO() after failing check\n \tok 14 - TEST_TODO() after failing check returns 0\n-\t# check \"0\" failed at t/unit-tests/t-basic.c:56\n+\t# check \"0\" failed at t/helper/test-example-tap.c:57\n \tnot ok 15 - failing check after TEST_TODO()\n \tok 16 - failing check after TEST_TODO() returns 0\n-\t# check \"!strcmp(\"\\thello\\\\\\\\\", \"there\\\"\\n\")\" failed at t/unit-tests/t-basic.c:61\n+\t# check \"!strcmp(\"\\thello\\\\\\\\\", \"there\\\"\\n\")\" failed at t/helper/test-example-tap.c:62\n \t# left: \"\\011hello\\\\\\\\\"\n \t# right: \"there\\\"\\012\"\n-\t# check \"!strcmp(\"NULL\", NULL)\" failed at t/unit-tests/t-basic.c:62\n+\t# check \"!strcmp(\"NULL\", NULL)\" failed at t/helper/test-example-tap.c:63\n \t# left: \"NULL\"\n \t# right: NULL\n-\t# check \"${SQ}a${SQ} == ${SQ}\\n${SQ}\" failed at t/unit-tests/t-basic.c:63\n+\t# check \"${SQ}a${SQ} == ${SQ}\\n${SQ}\" failed at t/helper/test-example-tap.c:64\n \t# left: ${SQ}a${SQ}\n \t# right: ${SQ}\\012${SQ}\n-\t# check \"${SQ}\\\\\\\\${SQ} == ${SQ}\\\\${SQ}${SQ}\" failed at t/unit-tests/t-basic.c:64\n+\t# check \"${SQ}\\\\\\\\${SQ} == ${SQ}\\\\${SQ}${SQ}\" failed at t/helper/test-example-tap.c:65\n \t# left: ${SQ}\\\\\\\\${SQ}\n \t# right: ${SQ}\\\\${SQ}${SQ}\n \tnot ok 17 - messages from failing string and char comparison\n-\t# BUG: test has no checks at t/unit-tests/t-basic.c:91\n+\t# BUG: test has no checks at t/helper/test-example-tap.c:92\n \tnot ok 18 - test with no checks\n \tok 19 - test with no checks returns 0\n \t1..19\n \tEOF\n \n-\t! \"$GIT_BUILD_DIR\"/t/unit-tests/bin/t-basic >actual &&\n+\t! test-tool example-tap >actual &&\n \ttest_cmp expect actual\n '\n \n-- \n2.43.0.594.gd9cf4e227d-goog\n\n"},{"id":"487854","messageId":"c8448406d71151714e89893208c46b8a4c369cb5.1706921262.git.steadmon@google.com","threadId":"60754","inReplyTo":"cover.1706921262.git.steadmon@google.com","subject":"[RFC PATCH v2 2/6] test-tool run-command testsuite: get shell from env","fromName":"Josh Steadmon","fromEmail":"steadmon@google.com","sentAt":"2024-02-03T00:50:27Z","receivedAt":"2024-02-03T00:50:39Z","isPatch":true,"sender":{"key":"steadmon@google.com","avatar":"https://avatars.githubusercontent.com/u/2654920?v=4"},"body":"When running tests through `test-tool run-command testsuite`, we\ncurrently hardcode `sh` as the command interpreter. As discussed in [1],\nthis is incorrect, and we should be using the shell set in\nTEST_SHELL_PATH instead.\n\nAdd a shell_path field in struct testsuite so that we can pass this to\nthe task runner callback. If this is non-null, we'll use it as the\nargv[0] of the subprocess. Otherwise, we'll just execute the test\nprogram directly.\n\nWhen setting up the struct testsuite in testsuite(), use the value\nof TEST_SHELL_PATH if it's set, otherwise default to `sh`.\n\n[1] https://lore.kernel.org/git/20240123005913.GB835964@coredump.intra.peff.net/\n\nSigned-off-by: Josh Steadmon \n---\n t/helper/test-run-command.c | 9 ++++++++-\n 1 file changed, 8 insertions(+), 1 deletion(-)\n\ndiff --git a/t/helper/test-run-command.c b/t/helper/test-run-command.c\nindex c0ed8722c8..a41a54d9cb 100644\n--- a/t/helper/test-run-command.c\n+++ b/t/helper/test-run-command.c\n@@ -65,6 +65,7 @@ struct testsuite {\n \tstruct string_list tests, failed;\n \tint next;\n \tint quiet, immediate, verbose, verbose_log, trace, write_junit_xml;\n+\tconst char *shell_path;\n };\n #define TESTSUITE_INIT { \\\n \t.tests = STRING_LIST_INIT_DUP, \\\n@@ -80,7 +81,9 @@ static int next_test(struct child_process *cp, struct strbuf *err, void *cb,\n \t\treturn 0;\n \n \ttest = suite->tests.items[suite->next++].string;\n-\tstrvec_pushl(&cp->args, \"sh\", test, NULL);\n+\tif (suite->shell_path)\n+\t\tstrvec_push(&cp->args, suite->shell_path);\n+\tstrvec_push(&cp->args, test);\n \tif (suite->quiet)\n \t\tstrvec_push(&cp->args, \"--quiet\");\n \tif (suite->immediate)\n@@ -162,6 +165,10 @@ static int testsuite(int argc, const char **argv)\n \tif (max_jobs <= 0)\n \t\tmax_jobs = online_cpus();\n \n+\tsuite.shell_path = getenv(\"TEST_SHELL_PATH\");\n+\tif (!suite.shell_path)\n+\t\tsuite.shell_path = \"sh\";\n+\n \tdir = opendir(\".\");\n \tif (!dir)\n \t\tdie(\"Could not open the current directory\");\n-- \n2.43.0.594.gd9cf4e227d-goog\n\n"},{"id":"487855","messageId":"e1b89ae93e930cd902d1527955d588c3d0c15490.1706921262.git.steadmon@google.com","threadId":"60754","inReplyTo":"cover.1706921262.git.steadmon@google.com","subject":"[RFC PATCH v2 3/6] test-tool run-command testsuite: remove hardcoded filter","fromName":"Josh Steadmon","fromEmail":"steadmon@google.com","sentAt":"2024-02-03T00:50:28Z","receivedAt":"2024-02-03T00:50:41Z","isPatch":true,"sender":{"key":"steadmon@google.com","avatar":"https://avatars.githubusercontent.com/u/2654920?v=4"},"body":"`test-tool run-command testsuite` currently assumes that it will only be\nrunning the shell test suite, and therefore filters out anything that\ndoes not match a hardcoded pattern of \"t[0-9][0-9][0-9][0-9]-*.sh\".\n\nLater in this series, we'll adapt `test-tool run-command testsuite` to\nalso support unit tests, which do not follow the same naming conventions\nas the shell tests, so this hardcoded pattern is inconvenient.\n\nSince `testsuite` also allows specifying patterns on the command-line,\nlet's just remove this pattern. As noted in [1], there are no longer any\nuses of `testsuite` in our codebase, it should be OK to break backwards\ncompatibility in this case. We also add a new filter to avoid trying to\nexecute \".\" and \"..\", so that users who wish to execute every test in a\ndirectory can do so without specifying a pattern.\n\n[1] https://lore.kernel.org/git/850ea42c-f103-68d5-896b-9120e2628686@gmx.de/\n\nSigned-off-by: Josh Steadmon \n---\n t/helper/test-run-command.c | 4 +---\n 1 file changed, 1 insertion(+), 3 deletions(-)\n\ndiff --git a/t/helper/test-run-command.c b/t/helper/test-run-command.c\nindex a41a54d9cb..e6bd792274 100644\n--- a/t/helper/test-run-command.c\n+++ b/t/helper/test-run-command.c\n@@ -175,9 +175,7 @@ static int testsuite(int argc, const char **argv)\n \twhile ((d = readdir(dir))) {\n \t\tconst char *p = d->d_name;\n \n-\t\tif (*p != 't' || !isdigit(p[1]) || !isdigit(p[2]) ||\n-\t\t !isdigit(p[3]) || !isdigit(p[4]) || p[5] != '-' ||\n-\t\t !ends_with(p, \".sh\"))\n+\t\tif (!strcmp(p, \".\") || !strcmp(p, \"..\"))\n \t\t\tcontinue;\n \n \t\t/* No pattern: match all */\n-- \n2.43.0.594.gd9cf4e227d-goog\n\n"},{"id":"487856","messageId":"b5665386b56df91fa5d95ee5b11288b5853546f0.1706921262.git.steadmon@google.com","threadId":"60754","inReplyTo":"cover.1706921262.git.steadmon@google.com","subject":"[RFC PATCH v2 4/6] test-tool run-command testsuite: support unit tests","fromName":"Josh Steadmon","fromEmail":"steadmon@google.com","sentAt":"2024-02-03T00:50:29Z","receivedAt":"2024-02-03T00:50:42Z","isPatch":true,"sender":{"key":"steadmon@google.com","avatar":"https://avatars.githubusercontent.com/u/2654920?v=4"},"body":"Teach the testsuite runner in `test-tool run-command testsuite` how to\nrun unit tests: if TEST_SHELL_PATH is not set, assume that we're running\nthe programs directly from CWD, rather than defaulting to \"sh\" as an\ninterpreter.\n\nWith this change, you can now use test-tool to run the unit tests:\n$ make\n$ cd t/unit-tests/bin\n$ ../../helper/test-tool run-command testsuite\n\nThis should be helpful on Windows to allow running tests without\nrequiring Perl (for `prove`), as discussed in [1] and [2].\n\nThis again breaks backwards compatibility, as it is now required to set\nTEST_SHELL_PATH properly for executing shell scripts, but again, as\nnoted in [2], there are no longer any such invocations in our codebase.\n\n[1] https://lore.kernel.org/git/nycvar.QRO.7.76.6.2109091323150.59@tvgsbejvaqbjf.bet/\n[2] https://lore.kernel.org/git/850ea42c-f103-68d5-896b-9120e2628686@gmx.de/\n\nSigned-off-by: Josh Steadmon \n---\n t/helper/test-run-command.c | 18 +++++++++++++++---\n 1 file changed, 15 insertions(+), 3 deletions(-)\n\ndiff --git a/t/helper/test-run-command.c b/t/helper/test-run-command.c\nindex e6bd792274..a0b8dc6fd7 100644\n--- a/t/helper/test-run-command.c\n+++ b/t/helper/test-run-command.c\n@@ -158,6 +158,8 @@ static int testsuite(int argc, const char **argv)\n \t\t.task_finished = test_finished,\n \t\t.data = &suite,\n \t};\n+\tstruct strbuf progpath = STRBUF_INIT;\n+\tsize_t path_prefix_len;\n \n \targc = parse_options(argc, argv, NULL, options,\n \t\t\ttestsuite_usage, PARSE_OPT_STOP_AT_NON_OPTION);\n@@ -165,9 +167,14 @@ static int testsuite(int argc, const char **argv)\n \tif (max_jobs <= 0)\n \t\tmax_jobs = online_cpus();\n \n+\t/*\n+\t * If we run without a shell, we have to provide the relative path to\n+\t * the executables.\n+\t */\n \tsuite.shell_path = getenv(\"TEST_SHELL_PATH\");\n \tif (!suite.shell_path)\n-\t\tsuite.shell_path = \"sh\";\n+\t\tstrbuf_addstr(&progpath, \"./\");\n+\tpath_prefix_len = progpath.len;\n \n \tdir = opendir(\".\");\n \tif (!dir)\n@@ -180,13 +187,17 @@ static int testsuite(int argc, const char **argv)\n \n \t\t/* No pattern: match all */\n \t\tif (!argc) {\n-\t\t\tstring_list_append(&suite.tests, p);\n+\t\t\tstrbuf_setlen(&progpath, path_prefix_len);\n+\t\t\tstrbuf_addstr(&progpath, p);\n+\t\t\tstring_list_append(&suite.tests, progpath.buf);\n \t\t\tcontinue;\n \t\t}\n \n \t\tfor (i = 0; i < argc; i++)\n \t\t\tif (!wildmatch(argv[i], p, 0)) {\n-\t\t\t\tstring_list_append(&suite.tests, p);\n+\t\t\t\tstrbuf_setlen(&progpath, path_prefix_len);\n+\t\t\t\tstrbuf_addstr(&progpath, p);\n+\t\t\t\tstring_list_append(&suite.tests, progpath.buf);\n \t\t\t\tbreak;\n \t\t\t}\n \t}\n@@ -213,6 +224,7 @@ static int testsuite(int argc, const char **argv)\n \n \tstring_list_clear(&suite.tests, 0);\n \tstring_list_clear(&suite.failed, 0);\n+\tstrbuf_release(&progpath);\n \n \treturn ret;\n }\n-- \n2.43.0.594.gd9cf4e227d-goog\n\n"},{"id":"487857","messageId":"f2746703d554d65d41afe0e41b1c9757427cda26.1706921262.git.steadmon@google.com","threadId":"60754","inReplyTo":"cover.1706921262.git.steadmon@google.com","subject":"[RFC PATCH v2 5/6] unit tests: add rule for running with test-tool","fromName":"Josh Steadmon","fromEmail":"steadmon@google.com","sentAt":"2024-02-03T00:50:30Z","receivedAt":"2024-02-03T00:50:44Z","isPatch":true,"sender":{"key":"steadmon@google.com","avatar":"https://avatars.githubusercontent.com/u/2654920?v=4"},"body":"In the previous commit, we added support in test-tool for running\ncollections of unit tests. Now, add rules in t/Makefile for running in\nthis way.\n\nThis new rule can be executed from the top-level Makefile via\n`make DEFAULT_UNIT_TEST_TARGET=unit-tests-test-tool unit-tests`, or by\nsetting DEFAULT_UNIT_TEST_TARGET in config.mak.\n\nSigned-off-by: Josh Steadmon \n---\n Makefile | 2 +-\n t/Makefile | 10 +++++++++-\n 2 files changed, 10 insertions(+), 2 deletions(-)\n\ndiff --git a/Makefile b/Makefile\nindex ba55d817ee..b0d1f04b4d 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -3870,5 +3870,5 @@ $(UNIT_TEST_PROGS): $(UNIT_TEST_BIN)/%$X: $(UNIT_TEST_DIR)/%.o $(UNIT_TEST_DIR)/\n \n .PHONY: build-unit-tests unit-tests\n build-unit-tests: $(UNIT_TEST_PROGS)\n-unit-tests: $(UNIT_TEST_PROGS)\n+unit-tests: $(UNIT_TEST_PROGS) t/helper/test-tool$X\n \t$(MAKE) -C t/ unit-tests\ndiff --git a/t/Makefile b/t/Makefile\nindex 1283c90c10..6e6316c29b 100644\n--- a/t/Makefile\n+++ b/t/Makefile\n@@ -45,6 +45,7 @@ CHAINLINTTESTS = $(sort $(patsubst chainlint/%.test,%,$(wildcard chainlint/*.tes\n CHAINLINT = '$(PERL_PATH_SQ)' chainlint.pl\n UNIT_TEST_SOURCES = $(wildcard unit-tests/t-*.c)\n UNIT_TESTS = $(patsubst unit-tests/%.c,unit-tests/bin/%$(X),$(UNIT_TEST_SOURCES))\n+UNIT_TESTS_NO_DIR = $(notdir $(UNIT_TESTS))\n \n # `test-chainlint` (which is a dependency of `test-lint`, `test` and `prove`)\n # checks all tests in all scripts via a single invocation, so tell individual\n@@ -72,7 +73,7 @@ $(T):\n $(UNIT_TESTS):\n \t@echo \"*** $@ ***\"; $@\n \n-.PHONY: unit-tests unit-tests-raw unit-tests-prove\n+.PHONY: unit-tests unit-tests-raw unit-tests-prove unit-tests-test-tool\n unit-tests: $(DEFAULT_UNIT_TEST_TARGET)\n \n unit-tests-raw: $(UNIT_TESTS)\n@@ -80,6 +81,13 @@ unit-tests-raw: $(UNIT_TESTS)\n unit-tests-prove:\n \t@echo \"*** prove - unit tests ***\"; $(PROVE) $(GIT_PROVE_OPTS) $(UNIT_TESTS)\n \n+unit-tests-test-tool:\n+\t@echo \"*** test-tool - unit tests **\"\n+\t( \\\n+\t\tcd unit-tests/bin && \\\n+\t\t../../helper/test-tool$X run-command testsuite $(UNIT_TESTS_NO_DIR)\\\n+\t)\n+\n pre-clean:\n \t$(RM) -r '$(TEST_RESULTS_DIRECTORY_SQ)'\n \n-- \n2.43.0.594.gd9cf4e227d-goog\n\n"},{"id":"487858","messageId":"cd7467a7bd51fbc01c999ee1bd7688770b1d11e5.1706921262.git.steadmon@google.com","threadId":"60754","inReplyTo":"cover.1706921262.git.steadmon@google.com","subject":"[RFC PATCH v2 6/6] t/Makefile: run unit tests alongside shell tests","fromName":"Josh Steadmon","fromEmail":"steadmon@google.com","sentAt":"2024-02-03T00:50:31Z","receivedAt":"2024-02-03T00:50:46Z","isPatch":true,"sender":{"key":"steadmon@google.com","avatar":"https://avatars.githubusercontent.com/u/2654920?v=4"},"body":"From: Jeff King \n\nAdd a wrapper script to allow `prove` to run both shell tests and unit\ntests from a single invocation. This avoids issues around running prove\ntwice in CI, as discussed in [1].\n\nAdditionally, this moves the unit tests into the main dev workflow, so\nthat errors can be spotted more quickly.\n\nNEEDS WORK: as discussed in previous commits in this series, there's a\ndesire to avoid `prove` specifically and (IIUC) unnecessary\nfork()/exec()ing in general on Windows. This change adds an extra exec()\nfor each shell and unit test execution, will that be a problem for\nWindows?\n\n[1] https://lore.kernel.org/git/pull.1613.git.1699894837844.gitgitgadget@gmail.com/\n\nSigned-off-by: Jeff King \nSigned-off-by: Josh Steadmon \n---\n t/Makefile | 2 +-\n t/run-test.sh | 13 +++++++++++++\n 2 files changed, 14 insertions(+), 1 deletion(-)\n create mode 100755 t/run-test.sh\n\ndiff --git a/t/Makefile b/t/Makefile\nindex 6e6316c29b..6a67fc22d7 100644\n--- a/t/Makefile\n+++ b/t/Makefile\n@@ -64,7 +64,7 @@ failed:\n \ttest -z \"$$failed\" || $(MAKE) $$failed\n \n prove: pre-clean check-chainlint $(TEST_LINT)\n-\t@echo \"*** prove ***\"; $(CHAINLINTSUPPRESS) $(PROVE) --exec '$(TEST_SHELL_PATH_SQ)' $(GIT_PROVE_OPTS) $(T) :: $(GIT_TEST_OPTS)\n+\t@echo \"*** prove (shell & unit tests) ***\"; $(CHAINLINTSUPPRESS) $(PROVE) --exec ./run-test.sh $(GIT_PROVE_OPTS) $(T) $(UNIT_TESTS) :: $(GIT_TEST_OPTS)\n \t$(MAKE) clean-except-prove-cache\n \n $(T):\ndiff --git a/t/run-test.sh b/t/run-test.sh\nnew file mode 100755\nindex 0000000000..c29fef48dc\n--- /dev/null\n+++ b/t/run-test.sh\n@@ -0,0 +1,13 @@\n+#!/bin/sh\n+\n+# A simple wrapper to run shell tests via TEST_SHELL_PATH,\n+# or exec unit tests directly.\n+\n+case \"$1\" in\n+*.sh)\n+\texec ${TEST_SHELL_PATH:-/bin/sh} \"$@\"\n+\t;;\n+*)\n+\texec \"$@\"\n+\t;;\n+esac\n-- \n2.43.0.594.gd9cf4e227d-goog\n\n"},{"id":"487879","messageId":"xmqqjznlqt1k.fsf@gitster.g","threadId":"60754","inReplyTo":"cover.1706921262.git.steadmon@google.com","subject":"Re: [RFC PATCH v2 0/6] test-tool: add unit test suite runner","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2024-02-03T18:52:39Z","receivedAt":"2024-02-03T18:52:50Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Josh Steadmon writes:\n\n> Please note: this series has been rebased onto jk/unit-tests-buildfix.\n>\n> For various reasons (see discussion at [1]) we would like an alternative\n> to `prove` for running test suites (including the unit tests) on\n> Windows.\n>\n> This series extends the existing `test-tool run-command testsuite` to\n> support running unit tests. In addition, it includes some small\n> cleanups:\n> * move t-basic out of the unit-tests directory\n> * don't hardcode the shell for running tests in `test-tool ... testsuite`\n> * don't hardcode a test name filter in `test-tool ... testsuite`\n> * add a test wrapper script to allow unit tests and the shell test suite\n> to run in a single `prove` process\n>\n> Some known remaining bits of work:\n> * We should investigate switching the Windows CI to use `test-tool`\n> instead of prove. However, Windows CI seems broken on\n> jk/unit-tests-buildfix, and I haven't had time to determine why.\n\nThanks to Dscho who figured this out, the jk/unit-tests-buildfix\ntopic in my tree has been updated to pass \"win test (n)\" jobs.\n\n> * We should determine whether it is confusing or otherwise harmful to\n> people's workflow to have the unit tests run in parallel with shell\n> tests when using prove as the default test target.\n\nI do not know much about \"confusing\" thing, but if the user\nallocates, say, 16 jobs to run tests in parallel, and one of them\ndrives the \"unit test suite runner\" that wants to do its own\nparallelism, we'd easily end up busting the resource limit the\nend-user desires. It does not necessarily mean that we should limit\nthe parallelism of \"unit test suite runner\" to 1 under prove, though.\n"},{"id":"487934","messageId":"4e2cbadd-1e0c-4526-a50f-9ba8600e7788@gmail.com","threadId":"60754","inReplyTo":"b5665386b56df91fa5d95ee5b11288b5853546f0.1706921262.git.steadmon@google.com","subject":"Re: [RFC PATCH v2 4/6] test-tool run-command testsuite: support unit tests","fromName":"","fromEmail":"phillip.wood123@gmail.com","sentAt":"2024-02-05T16:16:35Z","receivedAt":"2024-02-05T16:16:42Z","isPatch":true,"sender":{"key":"phillip.wood@dunelm.org.uk","avatar":null},"body":"Hi Josh\n\nOn 03/02/2024 00:50, Josh Steadmon wrote:\n> Teach the testsuite runner in `test-tool run-command testsuite` how to\n> run unit tests: if TEST_SHELL_PATH is not set, assume that we're running\n> the programs directly from CWD, rather than defaulting to \"sh\" as an\n> interpreter.\n\nJudging from the last patch in this series it seems likely that we'll \nwant to run unit tests and integration tests parallel. In which case it \nmight be better to look at the filename extension to decide whether to \nsh as an interpreter so that we can avoid having to use a wrapper \nscript. Then\n\n cd t\n helper/test-tool run-command testsuite 't[0-9]*.sh' 'unit-tests/bin/*'\n\nwould run the integration tests via \"sh\" and the unit-tests directly. \nWe'd need to figure out how to look for tests in both directories as \nwell though...\n\nBest Wishes\n\nPhillip\n\n> With this change, you can now use test-tool to run the unit tests:\n> $ make\n> $ cd t/unit-tests/bin\n> $ ../../helper/test-tool run-command testsuite\n> \n> This should be helpful on Windows to allow running tests without\n> requiring Perl (for `prove`), as discussed in [1] and [2].\n> \n> This again breaks backwards compatibility, as it is now required to set\n> TEST_SHELL_PATH properly for executing shell scripts, but again, as\n> noted in [2], there are no longer any such invocations in our codebase.\n> \n> [1] https://lore.kernel.org/git/nycvar.QRO.7.76.6.2109091323150.59@tvgsbejvaqbjf.bet/\n> [2] https://lore.kernel.org/git/850ea42c-f103-68d5-896b-9120e2628686@gmx.de/\n> \n> Signed-off-by: Josh Steadmon \n> ---\n> t/helper/test-run-command.c | 18 +++++++++++++++---\n> 1 file changed, 15 insertions(+), 3 deletions(-)\n> \n> diff --git a/t/helper/test-run-command.c b/t/helper/test-run-command.c\n> index e6bd792274..a0b8dc6fd7 100644\n> --- a/t/helper/test-run-command.c\n> +++ b/t/helper/test-run-command.c\n> @@ -158,6 +158,8 @@ static int testsuite(int argc, const char **argv)\n> \t\t.task_finished = test_finished,\n> \t\t.data = &suite,\n> \t};\n> +\tstruct strbuf progpath = STRBUF_INIT;\n> +\tsize_t path_prefix_len;\n> \n> \targc = parse_options(argc, argv, NULL, options,\n> \t\t\ttestsuite_usage, PARSE_OPT_STOP_AT_NON_OPTION);\n> @@ -165,9 +167,14 @@ static int testsuite(int argc, const char **argv)\n> \tif (max_jobs <= 0)\n> \t\tmax_jobs = online_cpus();\n> \n> +\t/*\n> +\t * If we run without a shell, we have to provide the relative path to\n> +\t * the executables.\n> +\t */\n> \tsuite.shell_path = getenv(\"TEST_SHELL_PATH\");\n> \tif (!suite.shell_path)\n> -\t\tsuite.shell_path = \"sh\";\n> +\t\tstrbuf_addstr(&progpath, \"./\");\n> +\tpath_prefix_len = progpath.len;\n> \n> \tdir = opendir(\".\");\n> \tif (!dir)\n> @@ -180,13 +187,17 @@ static int testsuite(int argc, const char **argv)\n> \n> \t\t/* No pattern: match all */\n> \t\tif (!argc) {\n> -\t\t\tstring_list_append(&suite.tests, p);\n> +\t\t\tstrbuf_setlen(&progpath, path_prefix_len);\n> +\t\t\tstrbuf_addstr(&progpath, p);\n> +\t\t\tstring_list_append(&suite.tests, progpath.buf);\n> \t\t\tcontinue;\n> \t\t}\n> \n> \t\tfor (i = 0; i < argc; i++)\n> \t\t\tif (!wildmatch(argv[i], p, 0)) {\n> -\t\t\t\tstring_list_append(&suite.tests, p);\n> +\t\t\t\tstrbuf_setlen(&progpath, path_prefix_len);\n> +\t\t\t\tstrbuf_addstr(&progpath, p);\n> +\t\t\t\tstring_list_append(&suite.tests, progpath.buf);\n> \t\t\t\tbreak;\n> \t\t\t}\n> \t}\n> @@ -213,6 +224,7 @@ static int testsuite(int argc, const char **argv)\n> \n> \tstring_list_clear(&suite.tests, 0);\n> \tstring_list_clear(&suite.failed, 0);\n> +\tstrbuf_release(&progpath);\n> \n> \treturn ret;\n> }\n"},{"id":"488170","messageId":"xmqqbk8s80hh.fsf@gitster.g","threadId":"60754","inReplyTo":"b5665386b56df91fa5d95ee5b11288b5853546f0.1706921262.git.steadmon@google.com","subject":"Re: [RFC PATCH v2 4/6] test-tool run-command testsuite: support unit tests","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2024-02-07T20:48:10Z","receivedAt":"2024-02-07T20:48:17Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Josh Steadmon writes:\n\n> Teach the testsuite runner in `test-tool run-command testsuite` how to\n> run unit tests: if TEST_SHELL_PATH is not set, assume that we're running\n> the programs directly from CWD, rather than defaulting to \"sh\" as an\n> interpreter.\n\nHmph. It sounds more like \"the run-command testsuite subcommand\nonly runs programs in the current directory\", not \"assume\" (which\nimplies there is a way to override the assumption). Not that the\nlimitation would hurt us in any way, though.\n\n> +\t/*\n> +\t * If we run without a shell, we have to provide the relative path to\n> +\t * the executables.\n> +\t */\n\nIt sounds more like \"If TEST_SHELL_PATH is not given, then we run\nthem in the current directory.\".\n\nIt is perfectly fine, because ...\n\n> \tsuite.shell_path = getenv(\"TEST_SHELL_PATH\");\n> \tif (!suite.shell_path)\n> -\t\tsuite.shell_path = \"sh\";\n> +\t\tstrbuf_addstr(&progpath, \"./\");\n> +\tpath_prefix_len = progpath.len;\n> \n> \tdir = opendir(\".\");\n> \tif (!dir)\n> @@ -180,13 +187,17 @@ static int testsuite(int argc, const char **argv)\n> ...\n> \t\tfor (i = 0; i < argc; i++)\n> \t\t\tif (!wildmatch(argv[i], p, 0)) {\n> -\t\t\t\tstring_list_append(&suite.tests, p);\n> +\t\t\t\tstrbuf_setlen(&progpath, path_prefix_len);\n> +\t\t\t\tstrbuf_addstr(&progpath, p);\n> +\t\t\t\tstring_list_append(&suite.tests, progpath.buf);\n> \t\t\t\tbreak;\n> \t\t\t}\n> \t}\n\n... this \"prefixing\" is done to a path discovered by readdir() from\na directory handle obtained by opendir(\".\"). If there were a way to\npass paths to executables directly, possibly as absolute paths, the\nunconditional prefixing of \"./\" would have been a problem, but we do\nnot have such a facility, so this should be OK.\n\n"},{"id":"488172","messageId":"xmqq7cjg80e1.fsf@gitster.g","threadId":"60754","inReplyTo":"f2746703d554d65d41afe0e41b1c9757427cda26.1706921262.git.steadmon@google.com","subject":"Re: [RFC PATCH v2 5/6] unit tests: add rule for running with test-tool","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2024-02-07T20:50:14Z","receivedAt":"2024-02-07T20:50:20Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Josh Steadmon writes:\n\n> +unit-tests-test-tool:\n> +\t@echo \"*** test-tool - unit tests **\"\n> +\t( \\\n> +\t\tcd unit-tests/bin && \\\n> +\t\t../../helper/test-tool$X run-command testsuite $(UNIT_TESTS_NO_DIR)\\\n> +\t)\n\nThis limits us to the tests that the Makefile knows about, which is\ngood. Of course, we might bust the command line length limit some\nday when we have too many unit test executables, but for now this is\nsimple, clean, and readable.\n\nThanks.\n\n\n"},{"id":"488174","messageId":"xmqqzfwc6lle.fsf@gitster.g","threadId":"60754","inReplyTo":"cd7467a7bd51fbc01c999ee1bd7688770b1d11e5.1706921262.git.steadmon@google.com","subject":"Re: [RFC PATCH v2 6/6] t/Makefile: run unit tests alongside shell tests","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2024-02-07T20:55:09Z","receivedAt":"2024-02-07T20:55:12Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Josh Steadmon writes:\n\n> From: Jeff King \n>\n> Add a wrapper script to allow `prove` to run both shell tests and unit\n> tests from a single invocation. This avoids issues around running prove\n> twice in CI, as discussed in [1].\n>\n> Additionally, this moves the unit tests into the main dev workflow, so\n> that errors can be spotted more quickly.\n>\n> NEEDS WORK: as discussed in previous commits in this series, there's a\n> desire to avoid `prove` specifically and (IIUC) unnecessary\n> fork()/exec()ing in general on Windows. This change adds an extra exec()\n> for each shell and unit test execution, will that be a problem for\n> Windows?\n>\n> [1] https://lore.kernel.org/git/pull.1613.git.1699894837844.gitgitgadget@gmail.com/\n>\n> Signed-off-by: Jeff King \n> Signed-off-by: Josh Steadmon \n> ---\n> t/Makefile | 2 +-\n> t/run-test.sh | 13 +++++++++++++\n> 2 files changed, 14 insertions(+), 1 deletion(-)\n> create mode 100755 t/run-test.sh\n>\n> diff --git a/t/Makefile b/t/Makefile\n> index 6e6316c29b..6a67fc22d7 100644\n> --- a/t/Makefile\n> +++ b/t/Makefile\n> @@ -64,7 +64,7 @@ failed:\n> \ttest -z \"$$failed\" || $(MAKE) $$failed\n> \n> prove: pre-clean check-chainlint $(TEST_LINT)\n> -\t@echo \"*** prove ***\"; $(CHAINLINTSUPPRESS) $(PROVE) --exec '$(TEST_SHELL_PATH_SQ)' $(GIT_PROVE_OPTS) $(T) :: $(GIT_TEST_OPTS)\n> +\t@echo \"*** prove (shell & unit tests) ***\"; $(CHAINLINTSUPPRESS) $(PROVE) --exec ./run-test.sh $(GIT_PROVE_OPTS) $(T) $(UNIT_TESTS) :: $(GIT_TEST_OPTS)\n> \t$(MAKE) clean-except-prove-cache\n> \n> $(T):\n> diff --git a/t/run-test.sh b/t/run-test.sh\n> new file mode 100755\n> index 0000000000..c29fef48dc\n> --- /dev/null\n> +++ b/t/run-test.sh\n> @@ -0,0 +1,13 @@\n> +#!/bin/sh\n> +\n> +# A simple wrapper to run shell tests via TEST_SHELL_PATH,\n> +# or exec unit tests directly.\n> +\n> +case \"$1\" in\n> +*.sh)\n> +\texec ${TEST_SHELL_PATH:-/bin/sh} \"$@\"\n> +\t;;\n> +*)\n> +\texec \"$@\"\n> +\t;;\n> +esac\n\nHmph. This penalizes the non-unit tests by doing an extra \"exec\",\nonce per program?\n\nOf course we cannot run two $(PROVE) invocations serially, one for\ndoing $(T) and the other for doing $(UNIT_TESTS)?\n\n\n"},{"id":"488175","messageId":"xmqqsf246ll7.fsf@gitster.g","threadId":"60754","inReplyTo":"da756b4bfb9d1ce0d1213d585e72acfbf667e2a2.1706921262.git.steadmon@google.com","subject":"Re: [RFC PATCH v2 1/6] t0080: turn t-basic unit test into a helper","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2024-02-07T20:55:16Z","receivedAt":"2024-02-07T20:55:25Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Josh Steadmon writes:\n\n> -UNIT_TEST_PROGRAMS = $(patsubst unit-tests/%.c,unit-tests/bin/%$(X),$(UNIT_TEST_SOURCES))\n> -UNIT_TESTS = $(sort $(filter-out unit-tests/bin/t-basic%,$(UNIT_TEST_PROGRAMS)))\n> +UNIT_TESTS = $(patsubst unit-tests/%.c,unit-tests/bin/%$(X),$(UNIT_TEST_SOURCES))\n\nNice that we no longer need the special casing.\n\n> diff --git a/t/unit-tests/t-basic.c b/t/helper/test-example-tap.c\n> similarity index 95%\n> rename from t/unit-tests/t-basic.c\n> rename to t/helper/test-example-tap.c\n> index fda1ae59a6..21e4848e78 100644\n> --- a/t/unit-tests/t-basic.c\n> +++ b/t/helper/test-example-tap.c\n> @@ -1,4 +1,5 @@\n> -#include \"test-lib.h\"\n> +#include \"t/unit-tests/test-lib.h\"\n> +#include \"test-tool.h\"\n\nAs the first thing both of these headers include is\n\"git-compat-util.h\", so the ordering should be safe either way,\nbecause everybody else in the directory seems to include\n\"test-tool.h\" before including headers that are specific to the\nsubsystem it is testing, and \"t/unit-tests/test-lib.h\" in this case\nis the header that is specific to the unit-test subsystem being\ntested, it may raise fewer eyebrows if we swapped the order of the\ninclusion here.\n\nOther than that, looks good to me.\n\nThanks.\n"},{"id":"488176","messageId":"xmqqle7w6lkt.fsf@gitster.g","threadId":"60754","inReplyTo":"c8448406d71151714e89893208c46b8a4c369cb5.1706921262.git.steadmon@google.com","subject":"Re: [RFC PATCH v2 2/6] test-tool run-command testsuite: get shell from env","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2024-02-07T20:55:30Z","receivedAt":"2024-02-07T20:55:35Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Josh Steadmon writes:\n\n> When running tests through `test-tool run-command testsuite`, we\n> currently hardcode `sh` as the command interpreter. As discussed in [1],\n> this is incorrect, and we should be using the shell set in\n> TEST_SHELL_PATH instead.\n>\n> Add a shell_path field in struct testsuite so that we can pass this to\n> the task runner callback. If this is non-null, we'll use it as the\n> argv[0] of the subprocess. Otherwise, we'll just execute the test\n> program directly.\n\nThat sounds nice in theory, but ...\n\n> When setting up the struct testsuite in testsuite(), use the value\n> of TEST_SHELL_PATH if it's set, otherwise default to `sh`.\n\n... this done in the testsuite() function, doesn't suite.shell_path\nalways gets some non-NULL value? Perhaps in a later step we will\nadd a mechanism to allow suite.shell_path to be NULL when we know\nwe are running an executable, or something?\n\nLeaving readers in a bit of suspense may, especially in a series\nthat is short like this, be a good technique to entice them to keep\nreading, perhaps, but anyway, if that is what is intended, a simple\n\"this feature is not used in this step, but we will take advantage\nof it soon in a later step\" would be a good idea.\n"},{"id":"488177","messageId":"xmqqeddo6lkk.fsf@gitster.g","threadId":"60754","inReplyTo":"e1b89ae93e930cd902d1527955d588c3d0c15490.1706921262.git.steadmon@google.com","subject":"Re: [RFC PATCH v2 3/6] test-tool run-command testsuite: remove hardcoded filter","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2024-02-07T20:55:39Z","receivedAt":"2024-02-07T20:55:44Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Josh Steadmon writes:\n\n> `test-tool run-command testsuite` currently assumes that it will only be\n> running the shell test suite, and therefore filters out anything that\n> does not match a hardcoded pattern of \"t[0-9][0-9][0-9][0-9]-*.sh\".\n>\n> Later in this series, we'll adapt `test-tool run-command testsuite` to\n> also support unit tests, which do not follow the same naming conventions\n> as the shell tests, so this hardcoded pattern is inconvenient.\n\nMakes sense to explain what future steps this prepares the codebase\nfor like this.\n\n> Since `testsuite` also allows specifying patterns on the command-line,\n> let's just remove this pattern. As noted in [1], there are no longer any\n> uses of `testsuite` in our codebase, it should be OK to break backwards\n> compatibility in this case. We also add a new filter to avoid trying to\n> execute \".\" and \"..\", so that users who wish to execute every test in a\n> directory can do so without specifying a pattern.\n\nAs we discussed in Peff's Makefile change that enumerates \"which are\nthe unit-test programs?\" Generally, $(wildcard) and readdir() to\nslurp everything in a directory, including stuff that is an\nuntracked cruft, is not an excellent idea.\n\nThis is not an end-user facing program and we are in full control of\nits input (most notably, \"which ones should we be running?\"), I do\nnot think it would be a huge issue, though.\n\n> [1] https://lore.kernel.org/git/850ea42c-f103-68d5-896b-9120e2628686@gmx.de/\n>\n> Signed-off-by: Josh Steadmon \n> ---\n> t/helper/test-run-command.c | 4 +---\n> 1 file changed, 1 insertion(+), 3 deletions(-)\n>\n> diff --git a/t/helper/test-run-command.c b/t/helper/test-run-command.c\n> index a41a54d9cb..e6bd792274 100644\n> --- a/t/helper/test-run-command.c\n> +++ b/t/helper/test-run-command.c\n> @@ -175,9 +175,7 @@ static int testsuite(int argc, const char **argv)\n> \twhile ((d = readdir(dir))) {\n> \t\tconst char *p = d->d_name;\n> \n> -\t\tif (*p != 't' || !isdigit(p[1]) || !isdigit(p[2]) ||\n> -\t\t !isdigit(p[3]) || !isdigit(p[4]) || p[5] != '-' ||\n> -\t\t !ends_with(p, \".sh\"))\n> +\t\tif (!strcmp(p, \".\") || !strcmp(p, \"..\"))\n> \t\t\tcontinue;\n> \n> \t\t/* No pattern: match all */\n"},{"id":"488181","messageId":"xmqqy1bw564p.fsf@gitster.g","threadId":"60754","inReplyTo":"cover.1706921262.git.steadmon@google.com","subject":"Re: [RFC PATCH v2 0/6] test-tool: add unit test suite runner","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2024-02-07T21:14:30Z","receivedAt":"2024-02-07T21:14:35Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Josh Steadmon writes:\n\n> Some known remaining bits of work:\n> * We should investigate switching the Windows CI to use `test-tool`\n> instead of prove. However, Windows CI seems broken on\n> jk/unit-tests-buildfix, and I haven't had time to determine why.\n> * We should determine whether it is confusing or otherwise harmful to\n> people's workflow to have the unit tests run in parallel with shell\n> tests when using prove as the default test target.\n\nThe first one seems to have been resolved. The latter can happen\nwhile it cooks, I would presume. If there is no other comments,\nlet's mark the topic for 'next'.\n\nThanks.\n\n"},{"id":"488194","messageId":"20240207224350.GA537799@coredump.intra.peff.net","threadId":"60754","inReplyTo":"xmqqzfwc6lle.fsf@gitster.g","subject":"Re: [RFC PATCH v2 6/6] t/Makefile: run unit tests alongside shell tests","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2024-02-07T22:43:50Z","receivedAt":"2024-02-07T22:43:51Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Wed, Feb 07, 2024 at 12:55:09PM -0800, Junio C Hamano wrote:\n\n> > +# A simple wrapper to run shell tests via TEST_SHELL_PATH,\n> > +# or exec unit tests directly.\n> > +\n> > +case \"$1\" in\n> > +*.sh)\n> > +\texec ${TEST_SHELL_PATH:-/bin/sh} \"$@\"\n> > +\t;;\n> > +*)\n> > +\texec \"$@\"\n> > +\t;;\n> > +esac\n> \n> Hmph. This penalizes the non-unit tests by doing an extra \"exec\",\n> once per program?\n\nIt does, but IMHO that is not likely to be a problem. It's once per\ntop-level script (so ~1000), and each of those scripts spawns hundreds\nor thousands of sub-commands. I didn't do any measurements, though.\n\nYou can extend \"prove\" with extra perl modules so that it makes the\ndistinction internally without the extra shell invocation. But when I\ntried to do it, I found it rather baroque and complicated (I can't\nremember if I succeeded but found it too gross, or just gave up halfway\nthrough trying).\n\n> Of course we cannot run two $(PROVE) invocations serially, one for\n> doing $(T) and the other for doing $(UNIT_TESTS)?\n\nNot if they share the same command-line options. If you use something\nlike \"--state=slow,save\", then the first run will write the list of all\ntests to \".prove\", and then the second will run every test mentioned in\n.prove (in addition to the unit-tests provided on the command-line).\n\nYou should be able to work around it by passing \"--statefile\". I _think_\nit might be OK to just do that unconditionally. Something like:\n\n prove --exec $(TEST_SHELL_PATH $(GIT_PROVE_OPTS) $(T) :: $(GIT_TEST_OPTS)\n prove --statefile=.prove-unit-tests $(GIT_PROVE_OPTS) $(UNIT_TESTS) :: $(GIT_TEST_OPTS)\n\nand then it's just a noop if GIT_PROVE_OPTS doesn't use --state. But I\nhaven't played with it myself.\n\n-Peff\n"},{"id":"488195","messageId":"20240207225802.GA538110@coredump.intra.peff.net","threadId":"60754","inReplyTo":"da756b4bfb9d1ce0d1213d585e72acfbf667e2a2.1706921262.git.steadmon@google.com","subject":"Re: [RFC PATCH v2 1/6] t0080: turn t-basic unit test into a helper","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2024-02-07T22:58:02Z","receivedAt":"2024-02-07T22:58:03Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Fri, Feb 02, 2024 at 04:50:26PM -0800, Josh Steadmon wrote:\n\n> This has the additional benefit that test harnesses seeking to run all\n> unit tests can find them with a simple glob of \"t/unit-tests/bin/t-*\",\n> with no exceptions needed. This will be important in a later patch where\n> we add support for running the unit tests via a test-tool subcommand.\n\nIs this last paragraph still accurate? I think in this rebased version\nof the series, we'll continue to use $(UNIT_TESTS) derived from the\nsource list rather than a glob in bin/.\n\n> --- a/t/Makefile\n> +++ b/t/Makefile\n> @@ -44,8 +44,7 @@ TINTEROP = $(sort $(wildcard interop/i[0-9][0-9][0-9][0-9]-*.sh))\n> CHAINLINTTESTS = $(sort $(patsubst chainlint/%.test,%,$(wildcard chainlint/*.test)))\n> CHAINLINT = '$(PERL_PATH_SQ)' chainlint.pl\n> UNIT_TEST_SOURCES = $(wildcard unit-tests/t-*.c)\n> -UNIT_TEST_PROGRAMS = $(patsubst unit-tests/%.c,unit-tests/bin/%$(X),$(UNIT_TEST_SOURCES))\n> -UNIT_TESTS = $(sort $(filter-out unit-tests/bin/t-basic%,$(UNIT_TEST_PROGRAMS)))\n> +UNIT_TESTS = $(patsubst unit-tests/%.c,unit-tests/bin/%$(X),$(UNIT_TEST_SOURCES))\n\nThis drops the intermediate UNIT_TEST_PROGRAMS, which makes sense. It\nwas only used to keep the long lines a bit more readable. But it also\ndrops the $(sort) call. Do we need to keep it?\n\nCertainly I'd think we want the contents of $(UNIT_TESTS) to be in a\ndeterministic order. Does the $(wildcard) function already return things\nin sorted order? I can't find any mention in the documention. It seems\nto do so for me in a simple test, but aae5239be2 (t/Makefile: Use $(sort\n...) explicitly where needed, 2011-09-04) argues otherwise.\n\nSo I think we probably want to keep it (or possibly move it onto the\nUNIT_TEST_SOURCES line, which keeps it close to the wildcard call).\n\n-Peff\n"},{"id":"488196","messageId":"xmqqle7vrh3v.fsf@gitster.g","threadId":"60754","inReplyTo":"20240207224350.GA537799@coredump.intra.peff.net","subject":"Re: [RFC PATCH v2 6/6] t/Makefile: run unit tests alongside shell tests","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2024-02-07T23:26:28Z","receivedAt":"2024-02-07T23:26:31Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Jeff King writes:\n\n> Not if they share the same command-line options. If you use something\n> like \"--state=slow,save\", then the first run will write the list of all\n> tests to \".prove\", and then the second will run every test mentioned in\n> .prove (in addition to the unit-tests provided on the command-line).\n>\n> You should be able to work around it by passing \"--statefile\". I _think_\n> it might be OK to just do that unconditionally. Something like:\n>\n> prove --exec $(TEST_SHELL_PATH $(GIT_PROVE_OPTS) $(T) :: $(GIT_TEST_OPTS)\n> prove --statefile=.prove-unit-tests $(GIT_PROVE_OPTS) $(UNIT_TESTS) :: $(GIT_TEST_OPTS)\n>\n> and then it's just a noop if GIT_PROVE_OPTS doesn't use --state. But I\n> haven't played with it myself.\n\nI do not think it warrants such complexity. The wrapper script is\nfine.\n"},{"id":"488197","messageId":"xmqqh6ijrf4o.fsf@gitster.g","threadId":"60754","inReplyTo":"20240207225802.GA538110@coredump.intra.peff.net","subject":"Re: [RFC PATCH v2 1/6] t0080: turn t-basic unit test into a helper","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2024-02-08T00:09:11Z","receivedAt":"2024-02-08T00:09:17Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Jeff King writes:\n\n> So I think we probably want to keep it (or possibly move it onto the\n> UNIT_TEST_SOURCES line, which keeps it close to the wildcard call).\n\nSensible.\n"},{"id":"488468","messageId":"ZcqCuseY4jeuKvKE@google.com","threadId":"60754","inReplyTo":"xmqqsf246ll7.fsf@gitster.g","subject":"Re: [RFC PATCH v2 1/6] t0080: turn t-basic unit test into a helper","fromName":"Josh Steadmon","fromEmail":"steadmon@google.com","sentAt":"2024-02-12T20:42:34Z","receivedAt":"2024-02-12T20:42:40Z","isPatch":true,"sender":{"key":"steadmon@google.com","avatar":"https://avatars.githubusercontent.com/u/2654920?v=4"},"body":"On 2024.02.07 12:55, Junio C Hamano wrote:\n> Josh Steadmon writes:\n> \n> > -UNIT_TEST_PROGRAMS = $(patsubst unit-tests/%.c,unit-tests/bin/%$(X),$(UNIT_TEST_SOURCES))\n> > -UNIT_TESTS = $(sort $(filter-out unit-tests/bin/t-basic%,$(UNIT_TEST_PROGRAMS)))\n> > +UNIT_TESTS = $(patsubst unit-tests/%.c,unit-tests/bin/%$(X),$(UNIT_TEST_SOURCES))\n> \n> Nice that we no longer need the special casing.\n> \n> > diff --git a/t/unit-tests/t-basic.c b/t/helper/test-example-tap.c\n> > similarity index 95%\n> > rename from t/unit-tests/t-basic.c\n> > rename to t/helper/test-example-tap.c\n> > index fda1ae59a6..21e4848e78 100644\n> > --- a/t/unit-tests/t-basic.c\n> > +++ b/t/helper/test-example-tap.c\n> > @@ -1,4 +1,5 @@\n> > -#include \"test-lib.h\"\n> > +#include \"t/unit-tests/test-lib.h\"\n> > +#include \"test-tool.h\"\n> \n> As the first thing both of these headers include is\n> \"git-compat-util.h\", so the ordering should be safe either way,\n> because everybody else in the directory seems to include\n> \"test-tool.h\" before including headers that are specific to the\n> subsystem it is testing, and \"t/unit-tests/test-lib.h\" in this case\n> is the header that is specific to the unit-test subsystem being\n> tested, it may raise fewer eyebrows if we swapped the order of the\n> inclusion here.\n> \n> Other than that, looks good to me.\n> \n> Thanks.\n\nFixed in V3.\n"},{"id":"488469","messageId":"ZcqFOVuR0sxFDDUv@google.com","threadId":"60754","inReplyTo":"20240207225802.GA538110@coredump.intra.peff.net","subject":"Re: [RFC PATCH v2 1/6] t0080: turn t-basic unit test into a helper","fromName":"Josh Steadmon","fromEmail":"steadmon@google.com","sentAt":"2024-02-12T20:53:13Z","receivedAt":"2024-02-12T20:53:18Z","isPatch":true,"sender":{"key":"steadmon@google.com","avatar":"https://avatars.githubusercontent.com/u/2654920?v=4"},"body":"On 2024.02.07 17:58, Jeff King wrote:\n> On Fri, Feb 02, 2024 at 04:50:26PM -0800, Josh Steadmon wrote:\n> \n> > This has the additional benefit that test harnesses seeking to run all\n> > unit tests can find them with a simple glob of \"t/unit-tests/bin/t-*\",\n> > with no exceptions needed. This will be important in a later patch where\n> > we add support for running the unit tests via a test-tool subcommand.\n> \n> Is this last paragraph still accurate? I think in this rebased version\n> of the series, we'll continue to use $(UNIT_TESTS) derived from the\n> source list rather than a glob in bin/.\n\nRight, thanks for the catch. Removed in V3.\n\n\n> > --- a/t/Makefile\n> > +++ b/t/Makefile\n> > @@ -44,8 +44,7 @@ TINTEROP = $(sort $(wildcard interop/i[0-9][0-9][0-9][0-9]-*.sh))\n> > CHAINLINTTESTS = $(sort $(patsubst chainlint/%.test,%,$(wildcard chainlint/*.test)))\n> > CHAINLINT = '$(PERL_PATH_SQ)' chainlint.pl\n> > UNIT_TEST_SOURCES = $(wildcard unit-tests/t-*.c)\n> > -UNIT_TEST_PROGRAMS = $(patsubst unit-tests/%.c,unit-tests/bin/%$(X),$(UNIT_TEST_SOURCES))\n> > -UNIT_TESTS = $(sort $(filter-out unit-tests/bin/t-basic%,$(UNIT_TEST_PROGRAMS)))\n> > +UNIT_TESTS = $(patsubst unit-tests/%.c,unit-tests/bin/%$(X),$(UNIT_TEST_SOURCES))\n> \n> This drops the intermediate UNIT_TEST_PROGRAMS, which makes sense. It\n> was only used to keep the long lines a bit more readable. But it also\n> drops the $(sort) call. Do we need to keep it?\n> \n> Certainly I'd think we want the contents of $(UNIT_TESTS) to be in a\n> deterministic order. Does the $(wildcard) function already return things\n> in sorted order? I can't find any mention in the documention. It seems\n> to do so for me in a simple test, but aae5239be2 (t/Makefile: Use $(sort\n> ...) explicitly where needed, 2011-09-04) argues otherwise.\n\nI see this line in the docs [1]: \"As with wildcard expansion in rules,\nthe results of the wildcard function are sorted\". GNU Make has restored\nthe sorted behavior of $(wildcard) since 2018 [2]. I'll leave the sort\noff for now, but if folks feel like we need to support older versions of\n`make`, I'll add it back.\n\n[1] https://www.gnu.org/software/make/manual/html_node/Wildcard-Function.html\n[2] https://savannah.gnu.org/bugs/index.php?52076\n"},{"id":"488470","messageId":"ZcqKbV37GDfNb4oF@google.com","threadId":"60754","inReplyTo":"4e2cbadd-1e0c-4526-a50f-9ba8600e7788@gmail.com","subject":"Re: [RFC PATCH v2 4/6] test-tool run-command testsuite: support unit tests","fromName":"Josh Steadmon","fromEmail":"steadmon@google.com","sentAt":"2024-02-12T21:15:25Z","receivedAt":"2024-02-12T21:15:31Z","isPatch":true,"sender":{"key":"steadmon@google.com","avatar":"https://avatars.githubusercontent.com/u/2654920?v=4"},"body":"On 2024.02.05 16:16, phillip.wood123@gmail.com wrote:\n> Hi Josh\n> \n> On 03/02/2024 00:50, Josh Steadmon wrote:\n> > Teach the testsuite runner in `test-tool run-command testsuite` how to\n> > run unit tests: if TEST_SHELL_PATH is not set, assume that we're running\n> > the programs directly from CWD, rather than defaulting to \"sh\" as an\n> > interpreter.\n> \n> Judging from the last patch in this series it seems likely that we'll want\n> to run unit tests and integration tests parallel. In which case it might be\n> better to look at the filename extension to decide whether to sh as an\n> interpreter so that we can avoid having to use a wrapper script. Then\n> \n> cd t\n> helper/test-tool run-command testsuite 't[0-9]*.sh' 'unit-tests/bin/*'\n> \n> would run the integration tests via \"sh\" and the unit-tests directly. We'd\n> need to figure out how to look for tests in both directories as well\n> though...\n\nAt the moment, I'm not planning on trying to make unit tests and shell\ntests run under the same test-tool process. If that is a valuable\nchange, hopefully the Windows / CI experts can use this series as a\nstarting point to make additional test-tool changes. However, I will\nprobably not spend any further time on this area.\n"},{"id":"488472","messageId":"xmqq34tx5q6o.fsf@gitster.g","threadId":"60754","inReplyTo":"ZcqFOVuR0sxFDDUv@google.com","subject":"Re: [RFC PATCH v2 1/6] t0080: turn t-basic unit test into a helper","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2024-02-12T21:27:11Z","receivedAt":"2024-02-12T21:27:14Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Josh Steadmon writes:\n\n> I see this line in the docs [1]: \"As with wildcard expansion in rules,\n> the results of the wildcard function are sorted\". GNU Make has restored\n> the sorted behavior of $(wildcard) since 2018 [2]. I'll leave the sort\n> off for now, but if folks feel like we need to support older versions of\n> `make`, I'll add it back.\n>\n> [1] https://www.gnu.org/software/make/manual/html_node/Wildcard-Function.html\n> [2] https://savannah.gnu.org/bugs/index.php?52076\n\nThanks for digging. I thought I was certain that woldcard is sorted\nand stable and was quite perplexed when I could not find the mention\nin a version of doc I had handy (\"\"\"This is Edition 0.75, last\nupdated 19 January 2020, of 'The GNU Make Manual', for GNU 'make'\nversion 4.3.\"\"\").\n\n\n\n"},{"id":"488474","messageId":"ZcqPLR95C0PJPWH_@google.com","threadId":"60754","inReplyTo":"xmqqle7w6lkt.fsf@gitster.g","subject":"Re: [RFC PATCH v2 2/6] test-tool run-command testsuite: get shell from env","fromName":"Josh Steadmon","fromEmail":"steadmon@google.com","sentAt":"2024-02-12T21:35:41Z","receivedAt":"2024-02-12T21:35:52Z","isPatch":true,"sender":{"key":"steadmon@google.com","avatar":"https://avatars.githubusercontent.com/u/2654920?v=4"},"body":"On 2024.02.07 12:55, Junio C Hamano wrote:\n> Josh Steadmon writes:\n> \n> > When running tests through `test-tool run-command testsuite`, we\n> > currently hardcode `sh` as the command interpreter. As discussed in [1],\n> > this is incorrect, and we should be using the shell set in\n> > TEST_SHELL_PATH instead.\n> >\n> > Add a shell_path field in struct testsuite so that we can pass this to\n> > the task runner callback. If this is non-null, we'll use it as the\n> > argv[0] of the subprocess. Otherwise, we'll just execute the test\n> > program directly.\n> \n> That sounds nice in theory, but ...\n> \n> > When setting up the struct testsuite in testsuite(), use the value\n> > of TEST_SHELL_PATH if it's set, otherwise default to `sh`.\n> \n> ... this done in the testsuite() function, doesn't suite.shell_path\n> always gets some non-NULL value? Perhaps in a later step we will\n> add a mechanism to allow suite.shell_path to be NULL when we know\n> we are running an executable, or something?\n> \n> Leaving readers in a bit of suspense may, especially in a series\n> that is short like this, be a good technique to entice them to keep\n> reading, perhaps, but anyway, if that is what is intended, a simple\n> \"this feature is not used in this step, but we will take advantage\n> of it soon in a later step\" would be a good idea.\n\nReworded in V3.\n"},{"id":"488477","messageId":"ZcqgMgFx97DbH94y@google.com","threadId":"60754","inReplyTo":"xmqqeddo6lkk.fsf@gitster.g","subject":"Re: [RFC PATCH v2 3/6] test-tool run-command testsuite: remove hardcoded filter","fromName":"Josh Steadmon","fromEmail":"steadmon@google.com","sentAt":"2024-02-12T22:48:18Z","receivedAt":"2024-02-12T22:48:24Z","isPatch":true,"sender":{"key":"steadmon@google.com","avatar":"https://avatars.githubusercontent.com/u/2654920?v=4"},"body":"On 2024.02.07 12:55, Junio C Hamano wrote:\n> Josh Steadmon writes:\n> \n> > `test-tool run-command testsuite` currently assumes that it will only be\n> > running the shell test suite, and therefore filters out anything that\n> > does not match a hardcoded pattern of \"t[0-9][0-9][0-9][0-9]-*.sh\".\n> >\n> > Later in this series, we'll adapt `test-tool run-command testsuite` to\n> > also support unit tests, which do not follow the same naming conventions\n> > as the shell tests, so this hardcoded pattern is inconvenient.\n> \n> Makes sense to explain what future steps this prepares the codebase\n> for like this.\n> \n> > Since `testsuite` also allows specifying patterns on the command-line,\n> > let's just remove this pattern. As noted in [1], there are no longer any\n> > uses of `testsuite` in our codebase, it should be OK to break backwards\n> > compatibility in this case. We also add a new filter to avoid trying to\n> > execute \".\" and \"..\", so that users who wish to execute every test in a\n> > directory can do so without specifying a pattern.\n> \n> As we discussed in Peff's Makefile change that enumerates \"which are\n> the unit-test programs?\" Generally, $(wildcard) and readdir() to\n> slurp everything in a directory, including stuff that is an\n> untracked cruft, is not an excellent idea.\n> \n> This is not an end-user facing program and we are in full control of\n> its input (most notably, \"which ones should we be running?\"), I do\n> not think it would be a huge issue, though.\n\nWould you prefer if I remove the default behavior of \"run everything in\nthe CWD\" and require passing in at least one filename filter?\n"},{"id":"488478","messageId":"ZcqgvRXG_rgl0lpP@google.com","threadId":"60754","inReplyTo":"xmqqjznlqt1k.fsf@gitster.g","subject":"Re: [RFC PATCH v2 0/6] test-tool: add unit test suite runner","fromName":"Josh Steadmon","fromEmail":"steadmon@google.com","sentAt":"2024-02-12T22:50:37Z","receivedAt":"2024-02-12T22:50:43Z","isPatch":true,"sender":{"key":"steadmon@google.com","avatar":"https://avatars.githubusercontent.com/u/2654920?v=4"},"body":"On 2024.02.03 10:52, Junio C Hamano wrote:\n> Josh Steadmon writes:\n> > * We should determine whether it is confusing or otherwise harmful to\n> > people's workflow to have the unit tests run in parallel with shell\n> > tests when using prove as the default test target.\n> \n> I do not know much about \"confusing\" thing, but if the user\n> allocates, say, 16 jobs to run tests in parallel, and one of them\n> drives the \"unit test suite runner\" that wants to do its own\n> parallelism, we'd easily end up busting the resource limit the\n> end-user desires. It does not necessarily mean that we should limit\n> the parallelism of \"unit test suite runner\" to 1 under prove, though.\n\nThe current `prove` helper script does not do any separate parallelism,\nso we are fine in that case, and the new test-tool runner does not\ncurrently support running both unit tests and shell tests in the same\nprocess, so we should be OK in either case.\n"},{"id":"488479","messageId":"xmqqo7cl47qf.fsf@gitster.g","threadId":"60754","inReplyTo":"ZcqgMgFx97DbH94y@google.com","subject":"Re: [RFC PATCH v2 3/6] test-tool run-command testsuite: remove hardcoded filter","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2024-02-12T22:51:04Z","receivedAt":"2024-02-12T22:51:07Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Josh Steadmon writes:\n\n>> As we discussed in Peff's Makefile change that enumerates \"which are\n>> the unit-test programs?\" Generally, $(wildcard) and readdir() to\n>> slurp everything in a directory, including stuff that is an\n>> untracked cruft, is not an excellent idea.\n>> \n>> This is not an end-user facing program and we are in full control of\n>> its input (most notably, \"which ones should we be running?\"), I do\n>> not think it would be a huge issue, though.\n>\n> Would you prefer if I remove the default behavior of \"run everything in\n> the CWD\" and require passing in at least one filename filter?\n\nNo preference either way at all.\n"},{"id":"488504","messageId":"20240213074118.GA2225494@coredump.intra.peff.net","threadId":"60754","inReplyTo":"xmqq34tx5q6o.fsf@gitster.g","subject":"Re: [RFC PATCH v2 1/6] t0080: turn t-basic unit test into a helper","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2024-02-13T07:41:18Z","receivedAt":"2024-02-13T07:41:28Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Mon, Feb 12, 2024 at 01:27:11PM -0800, Junio C Hamano wrote:\n\n> Josh Steadmon writes:\n> \n> > I see this line in the docs [1]: \"As with wildcard expansion in rules,\n> > the results of the wildcard function are sorted\". GNU Make has restored\n> > the sorted behavior of $(wildcard) since 2018 [2]. I'll leave the sort\n> > off for now, but if folks feel like we need to support older versions of\n> > `make`, I'll add it back.\n> >\n> > [1] https://www.gnu.org/software/make/manual/html_node/Wildcard-Function.html\n> > [2] https://savannah.gnu.org/bugs/index.php?52076\n> \n> Thanks for digging. I thought I was certain that woldcard is sorted\n> and stable and was quite perplexed when I could not find the mention\n> in a version of doc I had handy (\"\"\"This is Edition 0.75, last\n> updated 19 January 2020, of 'The GNU Make Manual', for GNU 'make'\n> version 4.3.\"\"\").\n\nLikewise (mine is the latest version in Debian unstable). The change to\nsort comes from their[1] eedea52a, which was in GNU make 4.2.90. But the\nmatching documentation change didn't happen until 5b993ae, which was\n4.3.90 in late 2021. So that explains the mystery.\n\nThose dates imply to me that we should keep the $(sort), though. Six\nyears is not so long in distro timescales, especially given that Debian\nunstable is on a 4-year-old version. (And if we did want to get rid of\nit, certainly we should do so consistently across the Makefile in a\nseparate patch).\n\n-Peff\n\n[1] commit ids are from https://git.savannah.gnu.org/git/make.git\n"},{"id":"488529","messageId":"016c01da5e8a$1ada8fc0$508faf40$@nexbridge.com","threadId":"60754","inReplyTo":"20240213074118.GA2225494@coredump.intra.peff.net","subject":"RE: [RFC PATCH v2 1/6] t0080: turn t-basic unit test into a helper","fromName":"","fromEmail":"rsbecker@nexbridge.com","sentAt":"2024-02-13T14:36:54Z","receivedAt":"2024-02-13T14:37:23Z","isPatch":true,"sender":{"key":"randall.becker@nexbridge.ca","avatar":"https://avatars.githubusercontent.com/u/28956764?v=4"},"body":"On Tuesday, February 13, 2024 2:41 AM, Peff wrote:\n>On Mon, Feb 12, 2024 at 01:27:11PM -0800, Junio C Hamano wrote:\n>\n>> Josh Steadmon writes:\n>>\n>> > I see this line in the docs [1]: \"As with wildcard expansion in\n>> > rules, the results of the wildcard function are sorted\". GNU Make\n>> > has restored the sorted behavior of $(wildcard) since 2018 [2]. I'll\n>> > leave the sort off for now, but if folks feel like we need to\n>> > support older versions of `make`, I'll add it back.\n>> >\n>> > [1]\n>> > https://www.gnu.org/software/make/manual/html_node/Wildcard-Function\n>> > .html [2] https://savannah.gnu.org/bugs/index.php?52076\n>>\n>> Thanks for digging. I thought I was certain that woldcard is sorted\n>> and stable and was quite perplexed when I could not find the mention\n>> in a version of doc I had handy (\"\"\"This is Edition 0.75, last updated\n>> 19 January 2020, of 'The GNU Make Manual', for GNU 'make'\n>> version 4.3.\"\"\").\n>\n>Likewise (mine is the latest version in Debian unstable). The change to sort comes\n>from their[1] eedea52a, which was in GNU make 4.2.90. But the matching\n>documentation change didn't happen until 5b993ae, which was\n>4.3.90 in late 2021. So that explains the mystery.\n>\n>Those dates imply to me that we should keep the $(sort), though. Six years is not so\n>long in distro timescales, especially given that Debian unstable is on a 4-year-old\n>version. (And if we did want to get rid of it, certainly we should do so consistently\n>across the Makefile in a separate patch).\n\nI am stuck on 4.2.1 and cannot get to 4.3.90 any time soon. Can you want on this? It will take us out unless we can suppress the $(sort)\n\nSincerely,\nRandall\n\n"},{"id":"488547","messageId":"xmqqh6ic6zpn.fsf@gitster.g","threadId":"60754","inReplyTo":"20240213074118.GA2225494@coredump.intra.peff.net","subject":"Re: [RFC PATCH v2 1/6] t0080: turn t-basic unit test into a helper","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2024-02-13T17:28:20Z","receivedAt":"2024-02-13T17:28:23Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Jeff King writes:\n\n> Likewise (mine is the latest version in Debian unstable). The change to\n> sort comes from their[1] eedea52a, which was in GNU make 4.2.90. But the\n> matching documentation change didn't happen until 5b993ae, which was\n> 4.3.90 in late 2021. So that explains the mystery.\n>\n> Those dates imply to me that we should keep the $(sort), though. Six\n> years is not so long in distro timescales, especially given that Debian\n> unstable is on a 4-year-old version. (And if we did want to get rid of\n> it, certainly we should do so consistently across the Makefile in a\n> separate patch).\n\nSounds sensible. The topic is expecting a reroll so I'll make sure\nI won't touch it until I see an update.\n\nThanks.\n"},{"id":"489190","messageId":"ZdffAnSOX87Vmp-V@google.com","threadId":"60754","inReplyTo":"20240213074118.GA2225494@coredump.intra.peff.net","subject":"Re: [RFC PATCH v2 1/6] t0080: turn t-basic unit test into a helper","fromName":"Josh Steadmon","fromEmail":"steadmon@google.com","sentAt":"2024-02-22T23:55:46Z","receivedAt":"2024-02-22T23:55:52Z","isPatch":true,"sender":{"key":"steadmon@google.com","avatar":"https://avatars.githubusercontent.com/u/2654920?v=4"},"body":"On 2024.02.13 02:41, Jeff King wrote:\n> On Mon, Feb 12, 2024 at 01:27:11PM -0800, Junio C Hamano wrote:\n> \n> > Josh Steadmon writes:\n> > \n> > > I see this line in the docs [1]: \"As with wildcard expansion in rules,\n> > > the results of the wildcard function are sorted\". GNU Make has restored\n> > > the sorted behavior of $(wildcard) since 2018 [2]. I'll leave the sort\n> > > off for now, but if folks feel like we need to support older versions of\n> > > `make`, I'll add it back.\n> > >\n> > > [1] https://www.gnu.org/software/make/manual/html_node/Wildcard-Function.html\n> > > [2] https://savannah.gnu.org/bugs/index.php?52076\n> > \n> > Thanks for digging. I thought I was certain that woldcard is sorted\n> > and stable and was quite perplexed when I could not find the mention\n> > in a version of doc I had handy (\"\"\"This is Edition 0.75, last\n> > updated 19 January 2020, of 'The GNU Make Manual', for GNU 'make'\n> > version 4.3.\"\"\").\n> \n> Likewise (mine is the latest version in Debian unstable). The change to\n> sort comes from their[1] eedea52a, which was in GNU make 4.2.90. But the\n> matching documentation change didn't happen until 5b993ae, which was\n> 4.3.90 in late 2021. So that explains the mystery.\n> \n> Those dates imply to me that we should keep the $(sort), though. Six\n> years is not so long in distro timescales, especially given that Debian\n> unstable is on a 4-year-old version. (And if we did want to get rid of\n> it, certainly we should do so consistently across the Makefile in a\n> separate patch).\n\nMakes sense, thanks for investigating. I've restored the $(sort) in V3.\n"},{"id":"489191","messageId":"Zdffgzf74eNeNBnF@google.com","threadId":"60754","inReplyTo":"016c01da5e8a$1ada8fc0$508faf40$@nexbridge.com","subject":"Re: [RFC PATCH v2 1/6] t0080: turn t-basic unit test into a helper","fromName":"Josh Steadmon","fromEmail":"steadmon@google.com","sentAt":"2024-02-22T23:57:55Z","receivedAt":"2024-02-22T23:58:00Z","isPatch":true,"sender":{"key":"steadmon@google.com","avatar":"https://avatars.githubusercontent.com/u/2654920?v=4"},"body":"On 2024.02.13 09:36, rsbecker@nexbridge.com wrote:\n> On Tuesday, February 13, 2024 2:41 AM, Peff wrote:\n> >On Mon, Feb 12, 2024 at 01:27:11PM -0800, Junio C Hamano wrote:\n> >\n> >> Josh Steadmon writes:\n> >>\n> >> > I see this line in the docs [1]: \"As with wildcard expansion in\n> >> > rules, the results of the wildcard function are sorted\". GNU Make\n> >> > has restored the sorted behavior of $(wildcard) since 2018 [2]. I'll\n> >> > leave the sort off for now, but if folks feel like we need to\n> >> > support older versions of `make`, I'll add it back.\n> >> >\n> >> > [1]\n> >> > https://www.gnu.org/software/make/manual/html_node/Wildcard-Function\n> >> > .html [2] https://savannah.gnu.org/bugs/index.php?52076\n> >>\n> >> Thanks for digging. I thought I was certain that woldcard is sorted\n> >> and stable and was quite perplexed when I could not find the mention\n> >> in a version of doc I had handy (\"\"\"This is Edition 0.75, last updated\n> >> 19 January 2020, of 'The GNU Make Manual', for GNU 'make'\n> >> version 4.3.\"\"\").\n> >\n> >Likewise (mine is the latest version in Debian unstable). The change to sort comes\n> >from their[1] eedea52a, which was in GNU make 4.2.90. But the matching\n> >documentation change didn't happen until 5b993ae, which was\n> >4.3.90 in late 2021. So that explains the mystery.\n> >\n> >Those dates imply to me that we should keep the $(sort), though. Six years is not so\n> >long in distro timescales, especially given that Debian unstable is on a 4-year-old\n> >version. (And if we did want to get rid of it, certainly we should do so consistently\n> >across the Makefile in a separate patch).\n> \n> I am stuck on 4.2.1 and cannot get to 4.3.90 any time soon. Can you\n> want on this? It will take us out unless we can suppress the $(sort)\n\nHi Randall,\n\nI'm not sure I follow here. The change in 4.2.90 is that wildcard\nexpansion becomes sorted by default again. So adding the $(sort) back\nshouldn't cause any problems in 4.2.1. Or did I misunderstand your\npoint?\n"},{"id":"489193","messageId":"005801da65ec$3a35e9f0$aea1bdd0$@nexbridge.com","threadId":"60754","inReplyTo":"Zdffgzf74eNeNBnF@google.com","subject":"RE: [RFC PATCH v2 1/6] t0080: turn t-basic unit test into a helper","fromName":"","fromEmail":"rsbecker@nexbridge.com","sentAt":"2024-02-23T00:06:55Z","receivedAt":"2024-02-23T00:07:24Z","isPatch":true,"sender":{"key":"randall.becker@nexbridge.ca","avatar":"https://avatars.githubusercontent.com/u/28956764?v=4"},"body":"On Thursday, February 22, 2024 6:58 PM, Josh Steadmon wrote:\n>On 2024.02.13 09:36, rsbecker@nexbridge.com wrote:\n>> On Tuesday, February 13, 2024 2:41 AM, Peff wrote:\n>> >On Mon, Feb 12, 2024 at 01:27:11PM -0800, Junio C Hamano wrote:\n>> >\n>> >> Josh Steadmon writes:\n>> >>\n>> >> > I see this line in the docs [1]: \"As with wildcard expansion in\n>> >> > rules, the results of the wildcard function are sorted\". GNU Make\n>> >> > has restored the sorted behavior of $(wildcard) since 2018 [2].\n>> >> > I'll leave the sort off for now, but if folks feel like we need\n>> >> > to support older versions of `make`, I'll add it back.\n>> >> >\n>> >> > [1]\n>> >> > https://www.gnu.org/software/make/manual/html_node/Wildcard-Funct\n>> >> > ion .html [2] https://savannah.gnu.org/bugs/index.php?52076\n>> >>\n>> >> Thanks for digging. I thought I was certain that woldcard is\n>> >> sorted and stable and was quite perplexed when I could not find the\n>> >> mention in a version of doc I had handy (\"\"\"This is Edition 0.75,\n>> >> last updated\n>> >> 19 January 2020, of 'The GNU Make Manual', for GNU 'make'\n>> >> version 4.3.\"\"\").\n>> >\n>> >Likewise (mine is the latest version in Debian unstable). The change\n>> >to sort comes from their[1] eedea52a, which was in GNU make 4.2.90.\n>> >But the matching documentation change didn't happen until 5b993ae,\n>> >which was\n>> >4.3.90 in late 2021. So that explains the mystery.\n>> >\n>> >Those dates imply to me that we should keep the $(sort), though. Six\n>> >years is not so long in distro timescales, especially given that\n>> >Debian unstable is on a 4-year-old version. (And if we did want to\n>> >get rid of it, certainly we should do so consistently across the\nMakefile in a\n>separate patch).\n>>\n>> I am stuck on 4.2.1 and cannot get to 4.3.90 any time soon. Can you\n>> want on this? It will take us out unless we can suppress the $(sort)\n>\n>Hi Randall,\n>\n>I'm not sure I follow here. The change in 4.2.90 is that wildcard expansion\nbecomes\n>sorted by default again. So adding the $(sort) back shouldn't cause any\nproblems in\n>4.2.1. Or did I misunderstand your point?\n\nI thought you were referring to a new ${sort} behaviour in 4.2.90. My bad.\n\n"},{"id":"489261","messageId":"ZdkgJMu3bddMmdbd@google.com","threadId":"60754","inReplyTo":"xmqqbk8s80hh.fsf@gitster.g","subject":"Re: [RFC PATCH v2 4/6] test-tool run-command testsuite: support unit tests","fromName":"Josh Steadmon","fromEmail":"steadmon@google.com","sentAt":"2024-02-23T22:45:56Z","receivedAt":"2024-02-23T22:46:03Z","isPatch":true,"sender":{"key":"steadmon@google.com","avatar":"https://avatars.githubusercontent.com/u/2654920?v=4"},"body":"On 2024.02.07 12:48, Junio C Hamano wrote:\n> Josh Steadmon writes:\n> \n> > Teach the testsuite runner in `test-tool run-command testsuite` how to\n> > run unit tests: if TEST_SHELL_PATH is not set, assume that we're running\n> > the programs directly from CWD, rather than defaulting to \"sh\" as an\n> > interpreter.\n> \n> Hmph. It sounds more like \"the run-command testsuite subcommand\n> only runs programs in the current directory\", not \"assume\" (which\n> implies there is a way to override the assumption). Not that the\n> limitation would hurt us in any way, though.\n> \n> > +\t/*\n> > +\t * If we run without a shell, we have to provide the relative path to\n> > +\t * the executables.\n> > +\t */\n> \n> It sounds more like \"If TEST_SHELL_PATH is not given, then we run\n> them in the current directory.\".\n\nReworded both of these in V3.\n"},{"id":"489262","messageId":"cover.1708728717.git.steadmon@google.com","threadId":"60754","inReplyTo":"cover.1705443632.git.steadmon@google.com","subject":"[PATCH v3 0/7] test-tool: add unit test suite runner","fromName":"Josh Steadmon","fromEmail":"steadmon@google.com","sentAt":"2024-02-23T23:33:49Z","receivedAt":"2024-02-23T23:33:59Z","isPatch":true,"sender":{"key":"steadmon@google.com","avatar":"https://avatars.githubusercontent.com/u/2654920?v=4"},"body":"Please note: this series has once again been rebased onto the latest\njk/unit-tests-buildfix.\n\nFor various reasons (see discussion at [1]) we would like an alternative\nto `prove` for running test suites (including the unit tests) on\nWindows.\n\n[1] https://lore.kernel.org/git/pull.1613.git.1699894837844.gitgitgadget@gmail.com/\n\nThis series extends the existing `test-tool run-command testsuite` to\nsupport running unit tests. In addition, it includes some small\ncleanups:\n* move t-basic out of the unit-tests directory\n* don't hardcode the shell for running tests in `test-tool ... testsuite`\n* don't hardcode a test name filter in `test-tool ... testsuite`\n* add a test wrapper script to allow unit tests and the shell test suite\n to run in a single `prove` process\n\nChanges in V3:\n* Added new patch (#7) to use the new test-tool runner for unit tests in\n Windows CI.\n* Restored the explicit sort call in t/Makefile, for backwards\n compatibility with older GNU Make versions.\n* Rebased the series on the latest jk/unit-tests-buildfix branch.\n* Fixed header include order in patch #1.\n* Removed a paragraph in patch #1's commit message that is obsolete now\n that we're building the list of test files from the sources rather\n than by globbing.\n* Added a note in patch #2 that setting a NULL suite.shell_path will be\n used in a later commit.\n* Clarified up some sloppy wording in commit messages and comments in\n t/helper/test-run-command.c.\n\nChanges in V2:\n* Rebased the series on the latest jk/unit-tests-buildfix branch.\n* Patch 1: move t-basic to a test-tool subcommand rather than a new\n executable under t/t0080/\n* New patch 2: get the shell path from TEST_SHELL_PATH in\n `test-tool run-command testsuite`\n* New patch 3: remove the hardcoded filename filter in\n `test-tool run-command testsuite`\n* Patch 4 (previously 2): simplified now that we no longer need to add\n any command-line flags to support unit tests\n* Patch 5 (previously 3): avoid trying to run cmake *.pdb files by using\n the unit test list built in the makefile in jk/unit-tests-buildfix.\n\n\nJeff King (1):\n t/Makefile: run unit tests alongside shell tests\n\nJosh Steadmon (6):\n t0080: turn t-basic unit test into a helper\n test-tool run-command testsuite: get shell from env\n test-tool run-command testsuite: remove hardcoded filter\n test-tool run-command testsuite: support unit tests\n unit tests: add rule for running with test-tool\n ci: use test-tool as unit test runner on Windows\n\n Makefile | 6 ++--\n ci/run-build-and-tests.sh | 2 --\n ci/run-test-slice.sh | 2 +-\n t/Makefile | 14 ++++++++--\n .../t-basic.c => helper/test-example-tap.c} | 5 ++--\n t/helper/test-run-command.c | 28 +++++++++++++++----\n t/helper/test-tool.c | 1 +\n t/helper/test-tool.h | 1 +\n t/run-test.sh | 13 +++++++++\n t/t0080-unit-test-output.sh | 24 ++++++++--------\n 10 files changed, 67 insertions(+), 29 deletions(-)\n rename t/{unit-tests/t-basic.c => helper/test-example-tap.c} (95%)\n create mode 100755 t/run-test.sh\n\nRange-diff against v2:\n1: da756b4bfb ! 1: 6777451100 t0080: turn t-basic unit test into a helper\n @@ Commit message\n t0080-unit-test-output.sh, so let's move it to\n t/helper/test-example-tap.c and adjust Makefiles as necessary.\n \n - This has the additional benefit that test harnesses seeking to run all\n - unit tests can find them with a simple glob of \"t/unit-tests/bin/t-*\",\n - with no exceptions needed. This will be important in a later patch where\n - we add support for running the unit tests via a test-tool subcommand.\n -\n \n ## Makefile ##\n @@ Makefile: perf: all\n \t$(QUIET_LINK)$(CC) $(ALL_CFLAGS) -o $@ $(ALL_LDFLAGS) $(filter %.o,$^) $(filter %.a,$^) $(LIBS)\n \n ## t/Makefile ##\n -@@ t/Makefile: TINTEROP = $(sort $(wildcard interop/i[0-9][0-9][0-9][0-9]-*.sh))\n - CHAINLINTTESTS = $(sort $(patsubst chainlint/%.test,%,$(wildcard chainlint/*.test)))\n +@@ t/Makefile: CHAINLINTTESTS = $(sort $(patsubst chainlint/%.test,%,$(wildcard chainlint/*.tes\n CHAINLINT = '$(PERL_PATH_SQ)' chainlint.pl\n UNIT_TEST_SOURCES = $(wildcard unit-tests/t-*.c)\n --UNIT_TEST_PROGRAMS = $(patsubst unit-tests/%.c,unit-tests/bin/%$(X),$(UNIT_TEST_SOURCES))\n + UNIT_TEST_PROGRAMS = $(patsubst unit-tests/%.c,unit-tests/bin/%$(X),$(UNIT_TEST_SOURCES))\n -UNIT_TESTS = $(sort $(filter-out unit-tests/bin/t-basic%,$(UNIT_TEST_PROGRAMS)))\n -+UNIT_TESTS = $(patsubst unit-tests/%.c,unit-tests/bin/%$(X),$(UNIT_TEST_SOURCES))\n ++UNIT_TESTS = $(sort $(UNIT_TEST_PROGRAMS))\n \n # `test-chainlint` (which is a dependency of `test-lint`, `test` and `prove`)\n # checks all tests in all scripts via a single invocation, so tell individual\n @@ t/Makefile: TINTEROP = $(sort $(wildcard interop/i[0-9][0-9][0-9][0-9]-*.sh))\n ## t/unit-tests/t-basic.c => t/helper/test-example-tap.c ##\n @@\n -#include \"test-lib.h\"\n -+#include \"t/unit-tests/test-lib.h\"\n +#include \"test-tool.h\"\n ++#include \"t/unit-tests/test-lib.h\"\n \n /*\n * The purpose of this \"unit test\" is to verify a few invariants of the unit\n2: c8448406d7 ! 2: 24f47f8fc7 test-tool run-command testsuite: get shell from env\n @@ Commit message\n Add a shell_path field in struct testsuite so that we can pass this to\n the task runner callback. If this is non-null, we'll use it as the\n argv[0] of the subprocess. Otherwise, we'll just execute the test\n - program directly.\n + program directly. We will use this feature in a later commit to enable\n + running binary executable unit tests.\n \n - When setting up the struct testsuite in testsuite(), use the value\n - of TEST_SHELL_PATH if it's set, otherwise default to `sh`.\n + However, for now when setting up the struct testsuite in testsuite(),\n + use the value of TEST_SHELL_PATH if it's set, otherwise keep the\n + original behavior by defaulting to `sh`.\n \n [1] https://lore.kernel.org/git/20240123005913.GB835964@coredump.intra.peff.net/\n \n3: e1b89ae93e = 3: 4a16a3ec24 test-tool run-command testsuite: remove hardcoded filter\n4: b5665386b5 ! 4: abc9a7afe8 test-tool run-command testsuite: support unit tests\n @@ Commit message\n test-tool run-command testsuite: support unit tests\n \n Teach the testsuite runner in `test-tool run-command testsuite` how to\n - run unit tests: if TEST_SHELL_PATH is not set, assume that we're running\n - the programs directly from CWD, rather than defaulting to \"sh\" as an\n - interpreter.\n + run unit tests: if TEST_SHELL_PATH is not set, run the programs directly\n + from CWD, rather than defaulting to \"sh\" as an interpreter.\n \n With this change, you can now use test-tool to run the unit tests:\n $ make\n @@ t/helper/test-run-command.c: static int testsuite(int argc, const char **argv)\n \t\tmax_jobs = online_cpus();\n \n +\t/*\n -+\t * If we run without a shell, we have to provide the relative path to\n -+\t * the executables.\n ++\t * If we run without a shell, execute the programs directly from CWD.\n +\t */\n \tsuite.shell_path = getenv(\"TEST_SHELL_PATH\");\n \tif (!suite.shell_path)\n5: f2746703d5 ! 5: a8bbff2c6b unit tests: add rule for running with test-tool\n @@ Makefile: $(UNIT_TEST_PROGS): $(UNIT_TEST_BIN)/%$X: $(UNIT_TEST_DIR)/%.o $(UNIT_\n \t$(MAKE) -C t/ unit-tests\n \n ## t/Makefile ##\n -@@ t/Makefile: CHAINLINTTESTS = $(sort $(patsubst chainlint/%.test,%,$(wildcard chainlint/*.tes\n - CHAINLINT = '$(PERL_PATH_SQ)' chainlint.pl\n +@@ t/Makefile: CHAINLINT = '$(PERL_PATH_SQ)' chainlint.pl\n UNIT_TEST_SOURCES = $(wildcard unit-tests/t-*.c)\n - UNIT_TESTS = $(patsubst unit-tests/%.c,unit-tests/bin/%$(X),$(UNIT_TEST_SOURCES))\n + UNIT_TEST_PROGRAMS = $(patsubst unit-tests/%.c,unit-tests/bin/%$(X),$(UNIT_TEST_SOURCES))\n + UNIT_TESTS = $(sort $(UNIT_TEST_PROGRAMS))\n +UNIT_TESTS_NO_DIR = $(notdir $(UNIT_TESTS))\n \n # `test-chainlint` (which is a dependency of `test-lint`, `test` and `prove`)\n6: cd7467a7bd ! 6: cfcc4bd427 t/Makefile: run unit tests alongside shell tests\n @@ Commit message\n twice in CI, as discussed in [1].\n \n Additionally, this moves the unit tests into the main dev workflow, so\n - that errors can be spotted more quickly.\n -\n - NEEDS WORK: as discussed in previous commits in this series, there's a\n - desire to avoid `prove` specifically and (IIUC) unnecessary\n - fork()/exec()ing in general on Windows. This change adds an extra exec()\n - for each shell and unit test execution, will that be a problem for\n - Windows?\n + that errors can be spotted more quickly. Accordingly, we remove the\n + separate unit tests step for Linux CI. (We leave the Windows CI\n + unit-test step as-is, because the sharding scheme there involves\n + selecting specific test files rather than running `make test`.)\n \n [1] https://lore.kernel.org/git/pull.1613.git.1699894837844.gitgitgadget@gmail.com/\n \n Signed-off-by: Jeff King \n \n + ## ci/run-build-and-tests.sh ##\n +@@ ci/run-build-and-tests.sh: if test -n \"$run_tests\"\n + then\n + \tgroup \"Run tests\" make test ||\n + \thandle_failed_tests\n +-\tgroup \"Run unit tests\" \\\n +-\t\tmake DEFAULT_UNIT_TEST_TARGET=unit-tests-prove unit-tests\n + fi\n + check_unignored_build_artifacts\n + \n +\n ## t/Makefile ##\n @@ t/Makefile: failed:\n \ttest -z \"$$failed\" || $(MAKE) $$failed\n-: ---------- > 7: cbf37e0ddc ci: use test-tool as unit test runner on Windows\n\nbase-commit: 4904a4d08cc085716df12ce713ae7ee3d5ecb75a\n-- \n2.44.0.rc0.258.g7320e95886-goog\n\n"},{"id":"489263","messageId":"677745110078501e611faf3d1facd1518e0453a2.1708728717.git.steadmon@google.com","threadId":"60754","inReplyTo":"cover.1708728717.git.steadmon@google.com","subject":"[PATCH v3 1/7] t0080: turn t-basic unit test into a helper","fromName":"Josh Steadmon","fromEmail":"steadmon@google.com","sentAt":"2024-02-23T23:33:50Z","receivedAt":"2024-02-23T23:34:01Z","isPatch":true,"sender":{"key":"steadmon@google.com","avatar":"https://avatars.githubusercontent.com/u/2654920?v=4"},"body":"While t/unit-tests/t-basic.c uses the unit-test framework added in\ne137fe3b29 (unit tests: add TAP unit test framework, 2023-11-09), it is\nnot a true unit test in that it intentionally fails in order to exercise\nvarious codepaths in the unit-test framework. Thus, we intentionally\nexclude it when running unit tests through the various t/Makefile\ntargets. Instead, it is executed by t0080-unit-test-output.sh, which\nverifies its output follows the TAP format expected for the various\npass, skip, or fail cases.\n\nAs such, it makes more sense for t-basic to be a helper item for\nt0080-unit-test-output.sh, so let's move it to\nt/helper/test-example-tap.c and adjust Makefiles as necessary.\n\nSigned-off-by: Josh Steadmon \n---\n Makefile | 4 ++--\n t/Makefile | 2 +-\n .../t-basic.c => helper/test-example-tap.c} | 5 ++--\n t/helper/test-tool.c | 1 +\n t/helper/test-tool.h | 1 +\n t/t0080-unit-test-output.sh | 24 +++++++++----------\n 6 files changed, 20 insertions(+), 17 deletions(-)\n rename t/{unit-tests/t-basic.c => helper/test-example-tap.c} (95%)\n\ndiff --git a/Makefile b/Makefile\nindex 23723367b8..ba55d817ee 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -802,6 +802,7 @@ TEST_BUILTINS_OBJS += test-dump-split-index.o\n TEST_BUILTINS_OBJS += test-dump-untracked-cache.o\n TEST_BUILTINS_OBJS += test-env-helper.o\n TEST_BUILTINS_OBJS += test-example-decorate.o\n+TEST_BUILTINS_OBJS += test-example-tap.o\n TEST_BUILTINS_OBJS += test-find-pack.o\n TEST_BUILTINS_OBJS += test-fsmonitor-client.o\n TEST_BUILTINS_OBJS += test-genrandom.o\n@@ -1338,7 +1339,6 @@ THIRD_PARTY_SOURCES += compat/regex/%\n THIRD_PARTY_SOURCES += sha1collisiondetection/%\n THIRD_PARTY_SOURCES += sha1dc/%\n \n-UNIT_TEST_PROGRAMS += t-basic\n UNIT_TEST_PROGRAMS += t-mem-pool\n UNIT_TEST_PROGRAMS += t-strbuf\n UNIT_TEST_PROGRAMS += t-ctype\n@@ -3218,7 +3218,7 @@ perf: all\n \n .PRECIOUS: $(TEST_OBJS)\n \n-t/helper/test-tool$X: $(patsubst %,t/helper/%,$(TEST_BUILTINS_OBJS))\n+t/helper/test-tool$X: $(patsubst %,t/helper/%,$(TEST_BUILTINS_OBJS)) $(UNIT_TEST_DIR)/test-lib.o\n \n t/helper/test-%$X: t/helper/test-%.o GIT-LDFLAGS $(GITLIBS) $(REFTABLE_TEST_LIB)\n \t$(QUIET_LINK)$(CC) $(ALL_CFLAGS) -o $@ $(ALL_LDFLAGS) $(filter %.o,$^) $(filter %.a,$^) $(LIBS)\ndiff --git a/t/Makefile b/t/Makefile\nindex 2d95046f26..4861edafe6 100644\n--- a/t/Makefile\n+++ b/t/Makefile\n@@ -48,7 +48,7 @@ CHAINLINTTESTS = $(sort $(patsubst chainlint/%.test,%,$(wildcard chainlint/*.tes\n CHAINLINT = '$(PERL_PATH_SQ)' chainlint.pl\n UNIT_TEST_SOURCES = $(wildcard unit-tests/t-*.c)\n UNIT_TEST_PROGRAMS = $(patsubst unit-tests/%.c,unit-tests/bin/%$(X),$(UNIT_TEST_SOURCES))\n-UNIT_TESTS = $(sort $(filter-out unit-tests/bin/t-basic%,$(UNIT_TEST_PROGRAMS)))\n+UNIT_TESTS = $(sort $(UNIT_TEST_PROGRAMS))\n \n # `test-chainlint` (which is a dependency of `test-lint`, `test` and `prove`)\n # checks all tests in all scripts via a single invocation, so tell individual\ndiff --git a/t/unit-tests/t-basic.c b/t/helper/test-example-tap.c\nsimilarity index 95%\nrename from t/unit-tests/t-basic.c\nrename to t/helper/test-example-tap.c\nindex fda1ae59a6..d072ad559f 100644\n--- a/t/unit-tests/t-basic.c\n+++ b/t/helper/test-example-tap.c\n@@ -1,4 +1,5 @@\n-#include \"test-lib.h\"\n+#include \"test-tool.h\"\n+#include \"t/unit-tests/test-lib.h\"\n \n /*\n * The purpose of this \"unit test\" is to verify a few invariants of the unit\n@@ -69,7 +70,7 @@ static void t_empty(void)\n \t; /* empty */\n }\n \n-int cmd_main(int argc, const char **argv)\n+int cmd__example_tap(int argc, const char **argv)\n {\n \ttest_res = TEST(check_res = check_int(1, ==, 1), \"passing test\");\n \tTEST(t_res(1), \"passing test and assertion return 1\");\ndiff --git a/t/helper/test-tool.c b/t/helper/test-tool.c\nindex 33b9501c21..bb5c04c9c0 100644\n--- a/t/helper/test-tool.c\n+++ b/t/helper/test-tool.c\n@@ -29,6 +29,7 @@ static struct test_cmd cmds[] = {\n \t{ \"dump-untracked-cache\", cmd__dump_untracked_cache },\n \t{ \"env-helper\", cmd__env_helper },\n \t{ \"example-decorate\", cmd__example_decorate },\n+\t{ \"example-tap\", cmd__example_tap },\n \t{ \"find-pack\", cmd__find_pack },\n \t{ \"fsmonitor-client\", cmd__fsmonitor_client },\n \t{ \"genrandom\", cmd__genrandom },\ndiff --git a/t/helper/test-tool.h b/t/helper/test-tool.h\nindex b72f07ded9..38001bd1c6 100644\n--- a/t/helper/test-tool.h\n+++ b/t/helper/test-tool.h\n@@ -23,6 +23,7 @@ int cmd__dump_untracked_cache(int argc, const char **argv);\n int cmd__dump_reftable(int argc, const char **argv);\n int cmd__env_helper(int argc, const char **argv);\n int cmd__example_decorate(int argc, const char **argv);\n+int cmd__example_tap(int argc, const char **argv);\n int cmd__find_pack(int argc, const char **argv);\n int cmd__fsmonitor_client(int argc, const char **argv);\n int cmd__genrandom(int argc, const char **argv);\ndiff --git a/t/t0080-unit-test-output.sh b/t/t0080-unit-test-output.sh\nindex 961b54b06c..83b1e3b7f5 100755\n--- a/t/t0080-unit-test-output.sh\n+++ b/t/t0080-unit-test-output.sh\n@@ -8,50 +8,50 @@ test_expect_success 'TAP output from unit tests' '\n \tcat >expect <<-EOF &&\n \tok 1 - passing test\n \tok 2 - passing test and assertion return 1\n-\t# check \"1 == 2\" failed at t/unit-tests/t-basic.c:76\n+\t# check \"1 == 2\" failed at t/helper/test-example-tap.c:77\n \t# left: 1\n \t# right: 2\n \tnot ok 3 - failing test\n \tok 4 - failing test and assertion return 0\n \tnot ok 5 - passing TEST_TODO() # TODO\n \tok 6 - passing TEST_TODO() returns 1\n-\t# todo check ${SQ}check(x)${SQ} succeeded at t/unit-tests/t-basic.c:25\n+\t# todo check ${SQ}check(x)${SQ} succeeded at t/helper/test-example-tap.c:26\n \tnot ok 7 - failing TEST_TODO()\n \tok 8 - failing TEST_TODO() returns 0\n-\t# check \"0\" failed at t/unit-tests/t-basic.c:30\n+\t# check \"0\" failed at t/helper/test-example-tap.c:31\n \t# skipping test - missing prerequisite\n-\t# skipping check ${SQ}1${SQ} at t/unit-tests/t-basic.c:32\n+\t# skipping check ${SQ}1${SQ} at t/helper/test-example-tap.c:33\n \tok 9 - test_skip() # SKIP\n \tok 10 - skipped test returns 1\n \t# skipping test - missing prerequisite\n \tok 11 - test_skip() inside TEST_TODO() # SKIP\n \tok 12 - test_skip() inside TEST_TODO() returns 1\n-\t# check \"0\" failed at t/unit-tests/t-basic.c:48\n+\t# check \"0\" failed at t/helper/test-example-tap.c:49\n \tnot ok 13 - TEST_TODO() after failing check\n \tok 14 - TEST_TODO() after failing check returns 0\n-\t# check \"0\" failed at t/unit-tests/t-basic.c:56\n+\t# check \"0\" failed at t/helper/test-example-tap.c:57\n \tnot ok 15 - failing check after TEST_TODO()\n \tok 16 - failing check after TEST_TODO() returns 0\n-\t# check \"!strcmp(\"\\thello\\\\\\\\\", \"there\\\"\\n\")\" failed at t/unit-tests/t-basic.c:61\n+\t# check \"!strcmp(\"\\thello\\\\\\\\\", \"there\\\"\\n\")\" failed at t/helper/test-example-tap.c:62\n \t# left: \"\\011hello\\\\\\\\\"\n \t# right: \"there\\\"\\012\"\n-\t# check \"!strcmp(\"NULL\", NULL)\" failed at t/unit-tests/t-basic.c:62\n+\t# check \"!strcmp(\"NULL\", NULL)\" failed at t/helper/test-example-tap.c:63\n \t# left: \"NULL\"\n \t# right: NULL\n-\t# check \"${SQ}a${SQ} == ${SQ}\\n${SQ}\" failed at t/unit-tests/t-basic.c:63\n+\t# check \"${SQ}a${SQ} == ${SQ}\\n${SQ}\" failed at t/helper/test-example-tap.c:64\n \t# left: ${SQ}a${SQ}\n \t# right: ${SQ}\\012${SQ}\n-\t# check \"${SQ}\\\\\\\\${SQ} == ${SQ}\\\\${SQ}${SQ}\" failed at t/unit-tests/t-basic.c:64\n+\t# check \"${SQ}\\\\\\\\${SQ} == ${SQ}\\\\${SQ}${SQ}\" failed at t/helper/test-example-tap.c:65\n \t# left: ${SQ}\\\\\\\\${SQ}\n \t# right: ${SQ}\\\\${SQ}${SQ}\n \tnot ok 17 - messages from failing string and char comparison\n-\t# BUG: test has no checks at t/unit-tests/t-basic.c:91\n+\t# BUG: test has no checks at t/helper/test-example-tap.c:92\n \tnot ok 18 - test with no checks\n \tok 19 - test with no checks returns 0\n \t1..19\n \tEOF\n \n-\t! \"$GIT_BUILD_DIR\"/t/unit-tests/bin/t-basic >actual &&\n+\t! test-tool example-tap >actual &&\n \ttest_cmp expect actual\n '\n \n-- \n2.44.0.rc0.258.g7320e95886-goog\n\n"},{"id":"489264","messageId":"24f47f8fc7e931c2e15fa985635e3036ec1123ba.1708728717.git.steadmon@google.com","threadId":"60754","inReplyTo":"cover.1708728717.git.steadmon@google.com","subject":"[PATCH v3 2/7] test-tool run-command testsuite: get shell from env","fromName":"Josh Steadmon","fromEmail":"steadmon@google.com","sentAt":"2024-02-23T23:33:51Z","receivedAt":"2024-02-23T23:34:02Z","isPatch":true,"sender":{"key":"steadmon@google.com","avatar":"https://avatars.githubusercontent.com/u/2654920?v=4"},"body":"When running tests through `test-tool run-command testsuite`, we\ncurrently hardcode `sh` as the command interpreter. As discussed in [1],\nthis is incorrect, and we should be using the shell set in\nTEST_SHELL_PATH instead.\n\nAdd a shell_path field in struct testsuite so that we can pass this to\nthe task runner callback. If this is non-null, we'll use it as the\nargv[0] of the subprocess. Otherwise, we'll just execute the test\nprogram directly. We will use this feature in a later commit to enable\nrunning binary executable unit tests.\n\nHowever, for now when setting up the struct testsuite in testsuite(),\nuse the value of TEST_SHELL_PATH if it's set, otherwise keep the\noriginal behavior by defaulting to `sh`.\n\n[1] https://lore.kernel.org/git/20240123005913.GB835964@coredump.intra.peff.net/\n\nSigned-off-by: Josh Steadmon \n---\n t/helper/test-run-command.c | 9 ++++++++-\n 1 file changed, 8 insertions(+), 1 deletion(-)\n\ndiff --git a/t/helper/test-run-command.c b/t/helper/test-run-command.c\nindex c0ed8722c8..a41a54d9cb 100644\n--- a/t/helper/test-run-command.c\n+++ b/t/helper/test-run-command.c\n@@ -65,6 +65,7 @@ struct testsuite {\n \tstruct string_list tests, failed;\n \tint next;\n \tint quiet, immediate, verbose, verbose_log, trace, write_junit_xml;\n+\tconst char *shell_path;\n };\n #define TESTSUITE_INIT { \\\n \t.tests = STRING_LIST_INIT_DUP, \\\n@@ -80,7 +81,9 @@ static int next_test(struct child_process *cp, struct strbuf *err, void *cb,\n \t\treturn 0;\n \n \ttest = suite->tests.items[suite->next++].string;\n-\tstrvec_pushl(&cp->args, \"sh\", test, NULL);\n+\tif (suite->shell_path)\n+\t\tstrvec_push(&cp->args, suite->shell_path);\n+\tstrvec_push(&cp->args, test);\n \tif (suite->quiet)\n \t\tstrvec_push(&cp->args, \"--quiet\");\n \tif (suite->immediate)\n@@ -162,6 +165,10 @@ static int testsuite(int argc, const char **argv)\n \tif (max_jobs <= 0)\n \t\tmax_jobs = online_cpus();\n \n+\tsuite.shell_path = getenv(\"TEST_SHELL_PATH\");\n+\tif (!suite.shell_path)\n+\t\tsuite.shell_path = \"sh\";\n+\n \tdir = opendir(\".\");\n \tif (!dir)\n \t\tdie(\"Could not open the current directory\");\n-- \n2.44.0.rc0.258.g7320e95886-goog\n\n"},{"id":"489265","messageId":"4a16a3ec241d83b3de424ce835db24eee6e46f0f.1708728717.git.steadmon@google.com","threadId":"60754","inReplyTo":"cover.1708728717.git.steadmon@google.com","subject":"[PATCH v3 3/7] test-tool run-command testsuite: remove hardcoded filter","fromName":"Josh Steadmon","fromEmail":"steadmon@google.com","sentAt":"2024-02-23T23:33:52Z","receivedAt":"2024-02-23T23:34:04Z","isPatch":true,"sender":{"key":"steadmon@google.com","avatar":"https://avatars.githubusercontent.com/u/2654920?v=4"},"body":"`test-tool run-command testsuite` currently assumes that it will only be\nrunning the shell test suite, and therefore filters out anything that\ndoes not match a hardcoded pattern of \"t[0-9][0-9][0-9][0-9]-*.sh\".\n\nLater in this series, we'll adapt `test-tool run-command testsuite` to\nalso support unit tests, which do not follow the same naming conventions\nas the shell tests, so this hardcoded pattern is inconvenient.\n\nSince `testsuite` also allows specifying patterns on the command-line,\nlet's just remove this pattern. As noted in [1], there are no longer any\nuses of `testsuite` in our codebase, it should be OK to break backwards\ncompatibility in this case. We also add a new filter to avoid trying to\nexecute \".\" and \"..\", so that users who wish to execute every test in a\ndirectory can do so without specifying a pattern.\n\n[1] https://lore.kernel.org/git/850ea42c-f103-68d5-896b-9120e2628686@gmx.de/\n\nSigned-off-by: Josh Steadmon \n---\n t/helper/test-run-command.c | 4 +---\n 1 file changed, 1 insertion(+), 3 deletions(-)\n\ndiff --git a/t/helper/test-run-command.c b/t/helper/test-run-command.c\nindex a41a54d9cb..e6bd792274 100644\n--- a/t/helper/test-run-command.c\n+++ b/t/helper/test-run-command.c\n@@ -175,9 +175,7 @@ static int testsuite(int argc, const char **argv)\n \twhile ((d = readdir(dir))) {\n \t\tconst char *p = d->d_name;\n \n-\t\tif (*p != 't' || !isdigit(p[1]) || !isdigit(p[2]) ||\n-\t\t !isdigit(p[3]) || !isdigit(p[4]) || p[5] != '-' ||\n-\t\t !ends_with(p, \".sh\"))\n+\t\tif (!strcmp(p, \".\") || !strcmp(p, \"..\"))\n \t\t\tcontinue;\n \n \t\t/* No pattern: match all */\n-- \n2.44.0.rc0.258.g7320e95886-goog\n\n"},{"id":"489266","messageId":"abc9a7afe8c1ab92bfbd2b9eea672bdd9ff3c107.1708728717.git.steadmon@google.com","threadId":"60754","inReplyTo":"cover.1708728717.git.steadmon@google.com","subject":"[PATCH v3 4/7] test-tool run-command testsuite: support unit tests","fromName":"Josh Steadmon","fromEmail":"steadmon@google.com","sentAt":"2024-02-23T23:33:53Z","receivedAt":"2024-02-23T23:34:06Z","isPatch":true,"sender":{"key":"steadmon@google.com","avatar":"https://avatars.githubusercontent.com/u/2654920?v=4"},"body":"Teach the testsuite runner in `test-tool run-command testsuite` how to\nrun unit tests: if TEST_SHELL_PATH is not set, run the programs directly\nfrom CWD, rather than defaulting to \"sh\" as an interpreter.\n\nWith this change, you can now use test-tool to run the unit tests:\n$ make\n$ cd t/unit-tests/bin\n$ ../../helper/test-tool run-command testsuite\n\nThis should be helpful on Windows to allow running tests without\nrequiring Perl (for `prove`), as discussed in [1] and [2].\n\nThis again breaks backwards compatibility, as it is now required to set\nTEST_SHELL_PATH properly for executing shell scripts, but again, as\nnoted in [2], there are no longer any such invocations in our codebase.\n\n[1] https://lore.kernel.org/git/nycvar.QRO.7.76.6.2109091323150.59@tvgsbejvaqbjf.bet/\n[2] https://lore.kernel.org/git/850ea42c-f103-68d5-896b-9120e2628686@gmx.de/\n\nSigned-off-by: Josh Steadmon \n---\n t/helper/test-run-command.c | 17 ++++++++++++++---\n 1 file changed, 14 insertions(+), 3 deletions(-)\n\ndiff --git a/t/helper/test-run-command.c b/t/helper/test-run-command.c\nindex e6bd792274..61eb1175fe 100644\n--- a/t/helper/test-run-command.c\n+++ b/t/helper/test-run-command.c\n@@ -158,6 +158,8 @@ static int testsuite(int argc, const char **argv)\n \t\t.task_finished = test_finished,\n \t\t.data = &suite,\n \t};\n+\tstruct strbuf progpath = STRBUF_INIT;\n+\tsize_t path_prefix_len;\n \n \targc = parse_options(argc, argv, NULL, options,\n \t\t\ttestsuite_usage, PARSE_OPT_STOP_AT_NON_OPTION);\n@@ -165,9 +167,13 @@ static int testsuite(int argc, const char **argv)\n \tif (max_jobs <= 0)\n \t\tmax_jobs = online_cpus();\n \n+\t/*\n+\t * If we run without a shell, execute the programs directly from CWD.\n+\t */\n \tsuite.shell_path = getenv(\"TEST_SHELL_PATH\");\n \tif (!suite.shell_path)\n-\t\tsuite.shell_path = \"sh\";\n+\t\tstrbuf_addstr(&progpath, \"./\");\n+\tpath_prefix_len = progpath.len;\n \n \tdir = opendir(\".\");\n \tif (!dir)\n@@ -180,13 +186,17 @@ static int testsuite(int argc, const char **argv)\n \n \t\t/* No pattern: match all */\n \t\tif (!argc) {\n-\t\t\tstring_list_append(&suite.tests, p);\n+\t\t\tstrbuf_setlen(&progpath, path_prefix_len);\n+\t\t\tstrbuf_addstr(&progpath, p);\n+\t\t\tstring_list_append(&suite.tests, progpath.buf);\n \t\t\tcontinue;\n \t\t}\n \n \t\tfor (i = 0; i < argc; i++)\n \t\t\tif (!wildmatch(argv[i], p, 0)) {\n-\t\t\t\tstring_list_append(&suite.tests, p);\n+\t\t\t\tstrbuf_setlen(&progpath, path_prefix_len);\n+\t\t\t\tstrbuf_addstr(&progpath, p);\n+\t\t\t\tstring_list_append(&suite.tests, progpath.buf);\n \t\t\t\tbreak;\n \t\t\t}\n \t}\n@@ -213,6 +223,7 @@ static int testsuite(int argc, const char **argv)\n \n \tstring_list_clear(&suite.tests, 0);\n \tstring_list_clear(&suite.failed, 0);\n+\tstrbuf_release(&progpath);\n \n \treturn ret;\n }\n-- \n2.44.0.rc0.258.g7320e95886-goog\n\n"},{"id":"489267","messageId":"a8bbff2c6bcea6a8874d2a145cffe370d6cbddfb.1708728717.git.steadmon@google.com","threadId":"60754","inReplyTo":"cover.1708728717.git.steadmon@google.com","subject":"[PATCH v3 5/7] unit tests: add rule for running with test-tool","fromName":"Josh Steadmon","fromEmail":"steadmon@google.com","sentAt":"2024-02-23T23:33:54Z","receivedAt":"2024-02-23T23:34:08Z","isPatch":true,"sender":{"key":"steadmon@google.com","avatar":"https://avatars.githubusercontent.com/u/2654920?v=4"},"body":"In the previous commit, we added support in test-tool for running\ncollections of unit tests. Now, add rules in t/Makefile for running in\nthis way.\n\nThis new rule can be executed from the top-level Makefile via\n`make DEFAULT_UNIT_TEST_TARGET=unit-tests-test-tool unit-tests`, or by\nsetting DEFAULT_UNIT_TEST_TARGET in config.mak.\n\nSigned-off-by: Josh Steadmon \n---\n Makefile | 2 +-\n t/Makefile | 10 +++++++++-\n 2 files changed, 10 insertions(+), 2 deletions(-)\n\ndiff --git a/Makefile b/Makefile\nindex ba55d817ee..b0d1f04b4d 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -3870,5 +3870,5 @@ $(UNIT_TEST_PROGS): $(UNIT_TEST_BIN)/%$X: $(UNIT_TEST_DIR)/%.o $(UNIT_TEST_DIR)/\n \n .PHONY: build-unit-tests unit-tests\n build-unit-tests: $(UNIT_TEST_PROGS)\n-unit-tests: $(UNIT_TEST_PROGS)\n+unit-tests: $(UNIT_TEST_PROGS) t/helper/test-tool$X\n \t$(MAKE) -C t/ unit-tests\ndiff --git a/t/Makefile b/t/Makefile\nindex 4861edafe6..0ae04f1e42 100644\n--- a/t/Makefile\n+++ b/t/Makefile\n@@ -49,6 +49,7 @@ CHAINLINT = '$(PERL_PATH_SQ)' chainlint.pl\n UNIT_TEST_SOURCES = $(wildcard unit-tests/t-*.c)\n UNIT_TEST_PROGRAMS = $(patsubst unit-tests/%.c,unit-tests/bin/%$(X),$(UNIT_TEST_SOURCES))\n UNIT_TESTS = $(sort $(UNIT_TEST_PROGRAMS))\n+UNIT_TESTS_NO_DIR = $(notdir $(UNIT_TESTS))\n \n # `test-chainlint` (which is a dependency of `test-lint`, `test` and `prove`)\n # checks all tests in all scripts via a single invocation, so tell individual\n@@ -76,7 +77,7 @@ $(T):\n $(UNIT_TESTS):\n \t@echo \"*** $@ ***\"; $@\n \n-.PHONY: unit-tests unit-tests-raw unit-tests-prove\n+.PHONY: unit-tests unit-tests-raw unit-tests-prove unit-tests-test-tool\n unit-tests: $(DEFAULT_UNIT_TEST_TARGET)\n \n unit-tests-raw: $(UNIT_TESTS)\n@@ -84,6 +85,13 @@ unit-tests-raw: $(UNIT_TESTS)\n unit-tests-prove:\n \t@echo \"*** prove - unit tests ***\"; $(PROVE) $(GIT_PROVE_OPTS) $(UNIT_TESTS)\n \n+unit-tests-test-tool:\n+\t@echo \"*** test-tool - unit tests **\"\n+\t( \\\n+\t\tcd unit-tests/bin && \\\n+\t\t../../helper/test-tool$X run-command testsuite $(UNIT_TESTS_NO_DIR)\\\n+\t)\n+\n pre-clean:\n \t$(RM) -r '$(TEST_RESULTS_DIRECTORY_SQ)'\n \n-- \n2.44.0.rc0.258.g7320e95886-goog\n\n"},{"id":"489268","messageId":"cfcc4bd427318fed1cacc8457381d5a0c408460a.1708728717.git.steadmon@google.com","threadId":"60754","inReplyTo":"cover.1708728717.git.steadmon@google.com","subject":"[PATCH v3 6/7] t/Makefile: run unit tests alongside shell tests","fromName":"Josh Steadmon","fromEmail":"steadmon@google.com","sentAt":"2024-02-23T23:33:55Z","receivedAt":"2024-02-23T23:34:10Z","isPatch":true,"sender":{"key":"steadmon@google.com","avatar":"https://avatars.githubusercontent.com/u/2654920?v=4"},"body":"From: Jeff King \n\nAdd a wrapper script to allow `prove` to run both shell tests and unit\ntests from a single invocation. This avoids issues around running prove\ntwice in CI, as discussed in [1].\n\nAdditionally, this moves the unit tests into the main dev workflow, so\nthat errors can be spotted more quickly. Accordingly, we remove the\nseparate unit tests step for Linux CI. (We leave the Windows CI\nunit-test step as-is, because the sharding scheme there involves\nselecting specific test files rather than running `make test`.)\n\n[1] https://lore.kernel.org/git/pull.1613.git.1699894837844.gitgitgadget@gmail.com/\n\nSigned-off-by: Jeff King \nSigned-off-by: Josh Steadmon \n---\n ci/run-build-and-tests.sh | 2 --\n t/Makefile | 2 +-\n t/run-test.sh | 13 +++++++++++++\n 3 files changed, 14 insertions(+), 3 deletions(-)\n create mode 100755 t/run-test.sh\n\ndiff --git a/ci/run-build-and-tests.sh b/ci/run-build-and-tests.sh\nindex 7a1466b868..2528f25e31 100755\n--- a/ci/run-build-and-tests.sh\n+++ b/ci/run-build-and-tests.sh\n@@ -50,8 +50,6 @@ if test -n \"$run_tests\"\n then\n \tgroup \"Run tests\" make test ||\n \thandle_failed_tests\n-\tgroup \"Run unit tests\" \\\n-\t\tmake DEFAULT_UNIT_TEST_TARGET=unit-tests-prove unit-tests\n fi\n check_unignored_build_artifacts\n \ndiff --git a/t/Makefile b/t/Makefile\nindex 0ae04f1e42..0c739754d8 100644\n--- a/t/Makefile\n+++ b/t/Makefile\n@@ -68,7 +68,7 @@ failed:\n \ttest -z \"$$failed\" || $(MAKE) $$failed\n \n prove: pre-clean check-chainlint $(TEST_LINT)\n-\t@echo \"*** prove ***\"; $(CHAINLINTSUPPRESS) $(PROVE) --exec '$(TEST_SHELL_PATH_SQ)' $(GIT_PROVE_OPTS) $(T) :: $(GIT_TEST_OPTS)\n+\t@echo \"*** prove (shell & unit tests) ***\"; $(CHAINLINTSUPPRESS) $(PROVE) --exec ./run-test.sh $(GIT_PROVE_OPTS) $(T) $(UNIT_TESTS) :: $(GIT_TEST_OPTS)\n \t$(MAKE) clean-except-prove-cache\n \n $(T):\ndiff --git a/t/run-test.sh b/t/run-test.sh\nnew file mode 100755\nindex 0000000000..c29fef48dc\n--- /dev/null\n+++ b/t/run-test.sh\n@@ -0,0 +1,13 @@\n+#!/bin/sh\n+\n+# A simple wrapper to run shell tests via TEST_SHELL_PATH,\n+# or exec unit tests directly.\n+\n+case \"$1\" in\n+*.sh)\n+\texec ${TEST_SHELL_PATH:-/bin/sh} \"$@\"\n+\t;;\n+*)\n+\texec \"$@\"\n+\t;;\n+esac\n-- \n2.44.0.rc0.258.g7320e95886-goog\n\n"},{"id":"489269","messageId":"cbf37e0ddc0e96361d9be60fce44fd3d250b1abf.1708728717.git.steadmon@google.com","threadId":"60754","inReplyTo":"cover.1708728717.git.steadmon@google.com","subject":"[PATCH v3 7/7] ci: use test-tool as unit test runner on Windows","fromName":"Josh Steadmon","fromEmail":"steadmon@google.com","sentAt":"2024-02-23T23:33:56Z","receivedAt":"2024-02-23T23:34:12Z","isPatch":true,"sender":{"key":"steadmon@google.com","avatar":"https://avatars.githubusercontent.com/u/2654920?v=4"},"body":"Although the previous commit changed t/Makefile to run unit tests\nalongside shell tests, the Windows CI still needs a separate unit-tests\nstep due to how the test sharding works.\n\nWe want to avoid using `prove` as a test running on Windows due to\nperformance issues [1], so use the new test-tool runner instead.\n\n[1] https://lore.kernel.org/git/850ea42c-f103-68d5-896b-9120e2628686@gmx.de/\n\nSigned-off-by: Josh Steadmon \n---\n ci/run-test-slice.sh | 2 +-\n 1 file changed, 1 insertion(+), 1 deletion(-)\n\ndiff --git a/ci/run-test-slice.sh b/ci/run-test-slice.sh\nindex ae8094382f..e167e646f7 100755\n--- a/ci/run-test-slice.sh\n+++ b/ci/run-test-slice.sh\n@@ -17,7 +17,7 @@ handle_failed_tests\n \n # We only have one unit test at the moment, so run it in the first slice\n if [ \"$1\" == \"0\" ] ; then\n-\tgroup \"Run unit tests\" make --quiet -C t unit-tests-prove\n+\tgroup \"Run unit tests\" make --quiet -C t unit-tests-test-tool\n fi\n \n check_unignored_build_artifacts\n-- \n2.44.0.rc0.258.g7320e95886-goog\n\n"},{"id":"491608","messageId":"xmqqjzlo8yqr.fsf@gitster.g","threadId":"60754","inReplyTo":"cover.1708728717.git.steadmon@google.com","subject":"Re: [PATCH v3 0/7] test-tool: add unit test suite runner","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2024-03-26T21:33:16Z","receivedAt":"2024-03-26T21:33:27Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Josh Steadmon writes:\n\n> Please note: this series has once again been rebased onto the latest\n> jk/unit-tests-buildfix.\n>\n> For various reasons (see discussion at [1]) we would like an alternative\n> to `prove` for running test suites (including the unit tests) on\n> Windows.\n\nFolks, what's the status of this one? I just checked the RFC thread\nand this last one (more than a month ago) and the issues seems to\nhave been addressed, but I prefer positive acks rather than \"we've\nseen it already, take our silence as the sign of endorsement\".\n\nThanks.\n"},{"id":"491672","messageId":"20240327085827.GA846805@coredump.intra.peff.net","threadId":"60754","inReplyTo":"cfcc4bd427318fed1cacc8457381d5a0c408460a.1708728717.git.steadmon@google.com","subject":"Re: [PATCH v3 6/7] t/Makefile: run unit tests alongside shell tests","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2024-03-27T08:58:27Z","receivedAt":"2024-03-27T08:58:29Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Fri, Feb 23, 2024 at 03:33:55PM -0800, Josh Steadmon wrote:\n\n> diff --git a/t/run-test.sh b/t/run-test.sh\n> new file mode 100755\n> index 0000000000..c29fef48dc\n> --- /dev/null\n> +++ b/t/run-test.sh\n> @@ -0,0 +1,13 @@\n> +#!/bin/sh\n> +\n> +# A simple wrapper to run shell tests via TEST_SHELL_PATH,\n> +# or exec unit tests directly.\n> +\n> +case \"$1\" in\n> +*.sh)\n> +\texec ${TEST_SHELL_PATH:-/bin/sh} \"$@\"\n> +\t;;\n> +*)\n> +\texec \"$@\"\n> +\t;;\n> +esac\n\nAn earlier step required that runs via \"test-tool run-command\" have\nTEST_SHELL_PATH set correctly. So defaulting to /bin/sh here is\npointless, I'd think? This is used only for the in-Makefile \"prove\"\ninvocation, so running individual tests or even a manual \"prove\" outside\nof the Makefile (where the user might not have set TEST_SHELL_PATH)\nwould not apply.\n\nIt obviously is not hurting anything, but I wonder if you'd want to have\nit complain loudly to catch any instance where your assumption is not\ntrue.\n\n-Peff\n"},{"id":"491673","messageId":"20240327090000.GB846805@coredump.intra.peff.net","threadId":"60754","inReplyTo":"xmqqjzlo8yqr.fsf@gitster.g","subject":"Re: [PATCH v3 0/7] test-tool: add unit test suite runner","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2024-03-27T09:00:00Z","receivedAt":"2024-03-27T09:00:02Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Tue, Mar 26, 2024 at 02:33:16PM -0700, Junio C Hamano wrote:\n\n> Josh Steadmon writes:\n> \n> > Please note: this series has once again been rebased onto the latest\n> > jk/unit-tests-buildfix.\n> >\n> > For various reasons (see discussion at [1]) we would like an alternative\n> > to `prove` for running test suites (including the unit tests) on\n> > Windows.\n> \n> Folks, what's the status of this one? I just checked the RFC thread\n> and this last one (more than a month ago) and the issues seems to\n> have been addressed, but I prefer positive acks rather than \"we've\n> seen it already, take our silence as the sign of endorsement\".\n\nIt is not a topic I am really interested in, so I was not following\nclosely. But I think my earlier issues were addressed, and from a quick\nread now it looks OK to me. (I left one small comment, but I don't think\nit's a big deal either way).\n\n-Peff\n"},{"id":"492794","messageId":"ZhgvefDKhTQ6rfnD@google.com","threadId":"60754","inReplyTo":"20240327085827.GA846805@coredump.intra.peff.net","subject":"Re: [PATCH v3 6/7] t/Makefile: run unit tests alongside shell tests","fromName":"Josh Steadmon","fromEmail":"steadmon@google.com","sentAt":"2024-04-11T18:44:09Z","receivedAt":"2024-04-11T18:44:16Z","isPatch":true,"sender":{"key":"steadmon@google.com","avatar":"https://avatars.githubusercontent.com/u/2654920?v=4"},"body":"On 2024.03.27 04:58, Jeff King wrote:\n> On Fri, Feb 23, 2024 at 03:33:55PM -0800, Josh Steadmon wrote:\n> \n> > diff --git a/t/run-test.sh b/t/run-test.sh\n> > new file mode 100755\n> > index 0000000000..c29fef48dc\n> > --- /dev/null\n> > +++ b/t/run-test.sh\n> > @@ -0,0 +1,13 @@\n> > +#!/bin/sh\n> > +\n> > +# A simple wrapper to run shell tests via TEST_SHELL_PATH,\n> > +# or exec unit tests directly.\n> > +\n> > +case \"$1\" in\n> > +*.sh)\n> > +\texec ${TEST_SHELL_PATH:-/bin/sh} \"$@\"\n> > +\t;;\n> > +*)\n> > +\texec \"$@\"\n> > +\t;;\n> > +esac\n> \n> An earlier step required that runs via \"test-tool run-command\" have\n> TEST_SHELL_PATH set correctly. So defaulting to /bin/sh here is\n> pointless, I'd think? This is used only for the in-Makefile \"prove\"\n> invocation, so running individual tests or even a manual \"prove\" outside\n> of the Makefile (where the user might not have set TEST_SHELL_PATH)\n> would not apply.\n\nActually, I think the \"manual prove outside of the Makefile\" situation\nis worth keeping this. I know I sometimes copy commands from Makefiles\nand run them manually when debugging issues, so it could be annoying for\nfolks if we remove the default here.\n\n> It obviously is not hurting anything, but I wonder if you'd want to have\n> it complain loudly to catch any instance where your assumption is not\n> true.\n> \n> -Peff\n"},{"id":"492828","messageId":"20240412042922.GB1077925@coredump.intra.peff.net","threadId":"60754","inReplyTo":"ZhgvefDKhTQ6rfnD@google.com","subject":"Re: [PATCH v3 6/7] t/Makefile: run unit tests alongside shell tests","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2024-04-12T04:29:22Z","receivedAt":"2024-04-12T04:29:23Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Thu, Apr 11, 2024 at 11:44:09AM -0700, Josh Steadmon wrote:\n\n> > An earlier step required that runs via \"test-tool run-command\" have\n> > TEST_SHELL_PATH set correctly. So defaulting to /bin/sh here is\n> > pointless, I'd think? This is used only for the in-Makefile \"prove\"\n> > invocation, so running individual tests or even a manual \"prove\" outside\n> > of the Makefile (where the user might not have set TEST_SHELL_PATH)\n> > would not apply.\n> \n> Actually, I think the \"manual prove outside of the Makefile\" situation\n> is worth keeping this. I know I sometimes copy commands from Makefiles\n> and run them manually when debugging issues, so it could be annoying for\n> folks if we remove the default here.\n\nHmm, by \"manually running prove\" I meant running:\n\n prove t0001-init.sh\n\nand so on. Or even \"prove --state=failed\" to re-run failed tests. But\nneither of those would even use this script, because there's no --exec\noption.\n\nBut it sounds like you mean literally cutting and pasting the \"prove\n--exec\" line from the Makefile. That seems to me like a weird thing to\nwant to do, but OK, I'll try not to judge your workflow. ;)\n\nBut if you are worried about making debugging more confusing, it seems\nlike silently defaulting to /bin/sh might make things worse. It is not\nnecessarily what \"make test\" did, and complaining loudly might be more\nhelpful than trying to run with an alternate shell.\n\nI don't feel too strongly about it, though. I'd generally just run\nsingle tests as \"./t0001-init.sh\", which runs into the same issue. I've\nbeen working on Git long enough that I know it is one of the possible\ngotchas when a test failure does not reproduce. :)\n\n-Peff\n"},{"id":"493425","messageId":"j3m7hw2tt5znk6ycriion5qn4usimoxseqv66n7wvpznwh774o@waaxau5hyt5y","threadId":"60754","inReplyTo":"20240412042922.GB1077925@coredump.intra.peff.net","subject":"Re: [PATCH v3 6/7] t/Makefile: run unit tests alongside shell tests","fromName":"Josh Steadmon","fromEmail":"steadmon@google.com","sentAt":"2024-04-24T18:57:42Z","receivedAt":"2024-04-24T18:57:48Z","isPatch":true,"sender":{"key":"steadmon@google.com","avatar":"https://avatars.githubusercontent.com/u/2654920?v=4"},"body":"On 2024.04.12 00:29, Jeff King wrote:\n> On Thu, Apr 11, 2024 at 11:44:09AM -0700, Josh Steadmon wrote:\n> \n> > > An earlier step required that runs via \"test-tool run-command\" have\n> > > TEST_SHELL_PATH set correctly. So defaulting to /bin/sh here is\n> > > pointless, I'd think? This is used only for the in-Makefile \"prove\"\n> > > invocation, so running individual tests or even a manual \"prove\" outside\n> > > of the Makefile (where the user might not have set TEST_SHELL_PATH)\n> > > would not apply.\n> > \n> > Actually, I think the \"manual prove outside of the Makefile\" situation\n> > is worth keeping this. I know I sometimes copy commands from Makefiles\n> > and run them manually when debugging issues, so it could be annoying for\n> > folks if we remove the default here.\n> \n> Hmm, by \"manually running prove\" I meant running:\n> \n> prove t0001-init.sh\n> \n> and so on. Or even \"prove --state=failed\" to re-run failed tests. But\n> neither of those would even use this script, because there's no --exec\n> option.\n> \n> But it sounds like you mean literally cutting and pasting the \"prove\n> --exec\" line from the Makefile. That seems to me like a weird thing to\n> want to do, but OK, I'll try not to judge your workflow. ;)\n> \n> But if you are worried about making debugging more confusing, it seems\n> like silently defaulting to /bin/sh might make things worse. It is not\n> necessarily what \"make test\" did, and complaining loudly might be more\n> helpful than trying to run with an alternate shell.\n> \n> I don't feel too strongly about it, though. I'd generally just run\n> single tests as \"./t0001-init.sh\", which runs into the same issue. I've\n> been working on Git long enough that I know it is one of the possible\n> gotchas when a test failure does not reproduce. :)\n> \n> -Peff\n\nAlright, fixed in V4. We now set TEST_SHELL_PATH when running `prove`,\nand we error out in run-test.sh if TEST_SHELL_PATH is not set.\n"},{"id":"493427","messageId":"cover.1713985716.git.steadmon@google.com","threadId":"60754","inReplyTo":"cover.1705443632.git.steadmon@google.com","subject":"[PATCH v4 0/7] test-tool: add unit test suite runner","fromName":"Josh Steadmon","fromEmail":"steadmon@google.com","sentAt":"2024-04-24T19:14:06Z","receivedAt":"2024-04-24T19:14:16Z","isPatch":true,"sender":{"key":"steadmon@google.com","avatar":"https://avatars.githubusercontent.com/u/2654920?v=4"},"body":"For various reasons (see discussion at [1]) we would like an alternative\nto `prove` for running test suites (including the unit tests) on\nWindows.\n\n[1] https://lore.kernel.org/git/pull.1613.git.1699894837844.gitgitgadget@gmail.com/\n\nThis series extends the existing `test-tool run-command testsuite` to\nsupport running unit tests. In addition, it includes some small\ncleanups:\n* move t-basic out of the unit-tests directory\n* don't hardcode the shell for running tests in `test-tool ... testsuite`\n* don't hardcode a test name filter in `test-tool ... testsuite`\n* add a test wrapper script to allow unit tests and the shell test suite\n to run in a single `prove` process\n\nChanges in V4:\n* Set TEST_SHELL_PATH when running the `prove` test target, and error\n out of run-test.sh if TEST_SHELL_PATH is not set.\n\nChanges in V3:\n* Added new patch (#7) to use the new test-tool runner for unit tests in\n Windows CI.\n* Restored the explicit sort call in t/Makefile, for backwards\n compatibility with older GNU Make versions.\n* Rebased the series on the latest jk/unit-tests-buildfix branch.\n* Fixed header include order in patch #1.\n* Removed a paragraph in patch #1's commit message that is obsolete now\n that we're building the list of test files from the sources rather\n than by globbing.\n* Added a note in patch #2 that setting a NULL suite.shell_path will be\n used in a later commit.\n* Clarified up some sloppy wording in commit messages and comments in\n t/helper/test-run-command.c.\n\nChanges in V2:\n* Rebased the series on the latest jk/unit-tests-buildfix branch.\n* Patch 1: move t-basic to a test-tool subcommand rather than a new\n executable under t/t0080/\n* New patch 2: get the shell path from TEST_SHELL_PATH in\n `test-tool run-command testsuite`\n* New patch 3: remove the hardcoded filename filter in\n `test-tool run-command testsuite`\n* Patch 4 (previously 2): simplified now that we no longer need to add\n any command-line flags to support unit tests\n* Patch 5 (previously 3): avoid trying to run cmake *.pdb files by using\n the unit test list built in the makefile in jk/unit-tests-buildfix.\n\n\nJeff King (1):\n t/Makefile: run unit tests alongside shell tests\n\nJosh Steadmon (6):\n t0080: turn t-basic unit test into a helper\n test-tool run-command testsuite: get shell from env\n test-tool run-command testsuite: remove hardcoded filter\n test-tool run-command testsuite: support unit tests\n unit tests: add rule for running with test-tool\n ci: use test-tool as unit test runner on Windows\n\n Makefile | 6 ++--\n ci/run-build-and-tests.sh | 2 --\n ci/run-test-slice.sh | 2 +-\n t/Makefile | 14 ++++++++--\n .../t-basic.c => helper/test-example-tap.c} | 5 ++--\n t/helper/test-run-command.c | 28 +++++++++++++++----\n t/helper/test-tool.c | 1 +\n t/helper/test-tool.h | 1 +\n t/run-test.sh | 17 +++++++++++\n t/t0080-unit-test-output.sh | 24 ++++++++--------\n 10 files changed, 71 insertions(+), 29 deletions(-)\n rename t/{unit-tests/t-basic.c => helper/test-example-tap.c} (95%)\n create mode 100755 t/run-test.sh\n\nRange-diff against v3:\n1: 6777451100 = 1: 6777451100 t0080: turn t-basic unit test into a helper\n2: 24f47f8fc7 = 2: 24f47f8fc7 test-tool run-command testsuite: get shell from env\n3: 4a16a3ec24 = 3: 4a16a3ec24 test-tool run-command testsuite: remove hardcoded filter\n4: abc9a7afe8 = 4: abc9a7afe8 test-tool run-command testsuite: support unit tests\n5: a8bbff2c6b = 5: a8bbff2c6b unit tests: add rule for running with test-tool\n6: cfcc4bd427 ! 6: 0e32de1afe t/Makefile: run unit tests alongside shell tests\n @@ t/Makefile: failed:\n\n prove: pre-clean check-chainlint $(TEST_LINT)\n - @echo \"*** prove ***\"; $(CHAINLINTSUPPRESS) $(PROVE) --exec '$(TEST_SHELL_PATH_SQ)' $(GIT_PROVE_OPTS) $(T) :: $(GIT_TEST_OPTS)\n -+ @echo \"*** prove (shell & unit tests) ***\"; $(CHAINLINTSUPPRESS) $(PROVE) --exec ./run-test.sh $(GIT_PROVE_OPTS) $(T) $(UNIT_TESTS) :: $(GIT_TEST_OPTS)\n ++ @echo \"*** prove (shell & unit tests) ***\"; $(CHAINLINTSUPPRESS) TEST_SHELL_PATH='$(TEST_SHELL_PATH_SQ)' $(PROVE) --exec ./run-test.sh $(GIT_PROVE_OPTS) $(T) $(UNIT_TESTS) :: $(GIT_TEST_OPTS)\n $(MAKE) clean-except-prove-cache\n\n $(T):\n @@ t/run-test.sh (new)\n +\n +case \"$1\" in\n +*.sh)\n -+ exec ${TEST_SHELL_PATH:-/bin/sh} \"$@\"\n ++ if test -z \"${TEST_SHELL_PATH+set}\" ; then\n ++ echo \"ERROR: TEST_SHELL_PATH is not set\" >&2\n ++ exit 1\n ++ fi\n ++ exec ${TEST_SHELL_PATH} \"$@\"\n + ;;\n +*)\n + exec \"$@\"\n7: cbf37e0ddc = 7: c562515293 ci: use test-tool as unit test runner on Windows\n\nbase-commit: 4904a4d08cc085716df12ce713ae7ee3d5ecb75a\n-- \n2.44.0.769.g3c40516874-goog\n\n"},{"id":"493428","messageId":"677745110078501e611faf3d1facd1518e0453a2.1713985716.git.steadmon@google.com","threadId":"60754","inReplyTo":"cover.1713985716.git.steadmon@google.com","subject":"[PATCH v4 1/7] t0080: turn t-basic unit test into a helper","fromName":"Josh Steadmon","fromEmail":"steadmon@google.com","sentAt":"2024-04-24T19:14:07Z","receivedAt":"2024-04-24T19:14:18Z","isPatch":true,"sender":{"key":"steadmon@google.com","avatar":"https://avatars.githubusercontent.com/u/2654920?v=4"},"body":"While t/unit-tests/t-basic.c uses the unit-test framework added in\ne137fe3b29 (unit tests: add TAP unit test framework, 2023-11-09), it is\nnot a true unit test in that it intentionally fails in order to exercise\nvarious codepaths in the unit-test framework. Thus, we intentionally\nexclude it when running unit tests through the various t/Makefile\ntargets. Instead, it is executed by t0080-unit-test-output.sh, which\nverifies its output follows the TAP format expected for the various\npass, skip, or fail cases.\n\nAs such, it makes more sense for t-basic to be a helper item for\nt0080-unit-test-output.sh, so let's move it to\nt/helper/test-example-tap.c and adjust Makefiles as necessary.\n\nSigned-off-by: Josh Steadmon \n---\n Makefile | 4 ++--\n t/Makefile | 2 +-\n .../t-basic.c => helper/test-example-tap.c} | 5 ++--\n t/helper/test-tool.c | 1 +\n t/helper/test-tool.h | 1 +\n t/t0080-unit-test-output.sh | 24 +++++++++----------\n 6 files changed, 20 insertions(+), 17 deletions(-)\n rename t/{unit-tests/t-basic.c => helper/test-example-tap.c} (95%)\n\ndiff --git a/Makefile b/Makefile\nindex 23723367b8..ba55d817ee 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -802,6 +802,7 @@ TEST_BUILTINS_OBJS += test-dump-split-index.o\n TEST_BUILTINS_OBJS += test-dump-untracked-cache.o\n TEST_BUILTINS_OBJS += test-env-helper.o\n TEST_BUILTINS_OBJS += test-example-decorate.o\n+TEST_BUILTINS_OBJS += test-example-tap.o\n TEST_BUILTINS_OBJS += test-find-pack.o\n TEST_BUILTINS_OBJS += test-fsmonitor-client.o\n TEST_BUILTINS_OBJS += test-genrandom.o\n@@ -1338,7 +1339,6 @@ THIRD_PARTY_SOURCES += compat/regex/%\n THIRD_PARTY_SOURCES += sha1collisiondetection/%\n THIRD_PARTY_SOURCES += sha1dc/%\n \n-UNIT_TEST_PROGRAMS += t-basic\n UNIT_TEST_PROGRAMS += t-mem-pool\n UNIT_TEST_PROGRAMS += t-strbuf\n UNIT_TEST_PROGRAMS += t-ctype\n@@ -3218,7 +3218,7 @@ perf: all\n \n .PRECIOUS: $(TEST_OBJS)\n \n-t/helper/test-tool$X: $(patsubst %,t/helper/%,$(TEST_BUILTINS_OBJS))\n+t/helper/test-tool$X: $(patsubst %,t/helper/%,$(TEST_BUILTINS_OBJS)) $(UNIT_TEST_DIR)/test-lib.o\n \n t/helper/test-%$X: t/helper/test-%.o GIT-LDFLAGS $(GITLIBS) $(REFTABLE_TEST_LIB)\n \t$(QUIET_LINK)$(CC) $(ALL_CFLAGS) -o $@ $(ALL_LDFLAGS) $(filter %.o,$^) $(filter %.a,$^) $(LIBS)\ndiff --git a/t/Makefile b/t/Makefile\nindex 2d95046f26..4861edafe6 100644\n--- a/t/Makefile\n+++ b/t/Makefile\n@@ -48,7 +48,7 @@ CHAINLINTTESTS = $(sort $(patsubst chainlint/%.test,%,$(wildcard chainlint/*.tes\n CHAINLINT = '$(PERL_PATH_SQ)' chainlint.pl\n UNIT_TEST_SOURCES = $(wildcard unit-tests/t-*.c)\n UNIT_TEST_PROGRAMS = $(patsubst unit-tests/%.c,unit-tests/bin/%$(X),$(UNIT_TEST_SOURCES))\n-UNIT_TESTS = $(sort $(filter-out unit-tests/bin/t-basic%,$(UNIT_TEST_PROGRAMS)))\n+UNIT_TESTS = $(sort $(UNIT_TEST_PROGRAMS))\n \n # `test-chainlint` (which is a dependency of `test-lint`, `test` and `prove`)\n # checks all tests in all scripts via a single invocation, so tell individual\ndiff --git a/t/unit-tests/t-basic.c b/t/helper/test-example-tap.c\nsimilarity index 95%\nrename from t/unit-tests/t-basic.c\nrename to t/helper/test-example-tap.c\nindex fda1ae59a6..d072ad559f 100644\n--- a/t/unit-tests/t-basic.c\n+++ b/t/helper/test-example-tap.c\n@@ -1,4 +1,5 @@\n-#include \"test-lib.h\"\n+#include \"test-tool.h\"\n+#include \"t/unit-tests/test-lib.h\"\n \n /*\n * The purpose of this \"unit test\" is to verify a few invariants of the unit\n@@ -69,7 +70,7 @@ static void t_empty(void)\n \t; /* empty */\n }\n \n-int cmd_main(int argc, const char **argv)\n+int cmd__example_tap(int argc, const char **argv)\n {\n \ttest_res = TEST(check_res = check_int(1, ==, 1), \"passing test\");\n \tTEST(t_res(1), \"passing test and assertion return 1\");\ndiff --git a/t/helper/test-tool.c b/t/helper/test-tool.c\nindex 33b9501c21..bb5c04c9c0 100644\n--- a/t/helper/test-tool.c\n+++ b/t/helper/test-tool.c\n@@ -29,6 +29,7 @@ static struct test_cmd cmds[] = {\n \t{ \"dump-untracked-cache\", cmd__dump_untracked_cache },\n \t{ \"env-helper\", cmd__env_helper },\n \t{ \"example-decorate\", cmd__example_decorate },\n+\t{ \"example-tap\", cmd__example_tap },\n \t{ \"find-pack\", cmd__find_pack },\n \t{ \"fsmonitor-client\", cmd__fsmonitor_client },\n \t{ \"genrandom\", cmd__genrandom },\ndiff --git a/t/helper/test-tool.h b/t/helper/test-tool.h\nindex b72f07ded9..38001bd1c6 100644\n--- a/t/helper/test-tool.h\n+++ b/t/helper/test-tool.h\n@@ -23,6 +23,7 @@ int cmd__dump_untracked_cache(int argc, const char **argv);\n int cmd__dump_reftable(int argc, const char **argv);\n int cmd__env_helper(int argc, const char **argv);\n int cmd__example_decorate(int argc, const char **argv);\n+int cmd__example_tap(int argc, const char **argv);\n int cmd__find_pack(int argc, const char **argv);\n int cmd__fsmonitor_client(int argc, const char **argv);\n int cmd__genrandom(int argc, const char **argv);\ndiff --git a/t/t0080-unit-test-output.sh b/t/t0080-unit-test-output.sh\nindex 961b54b06c..83b1e3b7f5 100755\n--- a/t/t0080-unit-test-output.sh\n+++ b/t/t0080-unit-test-output.sh\n@@ -8,50 +8,50 @@ test_expect_success 'TAP output from unit tests' '\n \tcat >expect <<-EOF &&\n \tok 1 - passing test\n \tok 2 - passing test and assertion return 1\n-\t# check \"1 == 2\" failed at t/unit-tests/t-basic.c:76\n+\t# check \"1 == 2\" failed at t/helper/test-example-tap.c:77\n \t# left: 1\n \t# right: 2\n \tnot ok 3 - failing test\n \tok 4 - failing test and assertion return 0\n \tnot ok 5 - passing TEST_TODO() # TODO\n \tok 6 - passing TEST_TODO() returns 1\n-\t# todo check ${SQ}check(x)${SQ} succeeded at t/unit-tests/t-basic.c:25\n+\t# todo check ${SQ}check(x)${SQ} succeeded at t/helper/test-example-tap.c:26\n \tnot ok 7 - failing TEST_TODO()\n \tok 8 - failing TEST_TODO() returns 0\n-\t# check \"0\" failed at t/unit-tests/t-basic.c:30\n+\t# check \"0\" failed at t/helper/test-example-tap.c:31\n \t# skipping test - missing prerequisite\n-\t# skipping check ${SQ}1${SQ} at t/unit-tests/t-basic.c:32\n+\t# skipping check ${SQ}1${SQ} at t/helper/test-example-tap.c:33\n \tok 9 - test_skip() # SKIP\n \tok 10 - skipped test returns 1\n \t# skipping test - missing prerequisite\n \tok 11 - test_skip() inside TEST_TODO() # SKIP\n \tok 12 - test_skip() inside TEST_TODO() returns 1\n-\t# check \"0\" failed at t/unit-tests/t-basic.c:48\n+\t# check \"0\" failed at t/helper/test-example-tap.c:49\n \tnot ok 13 - TEST_TODO() after failing check\n \tok 14 - TEST_TODO() after failing check returns 0\n-\t# check \"0\" failed at t/unit-tests/t-basic.c:56\n+\t# check \"0\" failed at t/helper/test-example-tap.c:57\n \tnot ok 15 - failing check after TEST_TODO()\n \tok 16 - failing check after TEST_TODO() returns 0\n-\t# check \"!strcmp(\"\\thello\\\\\\\\\", \"there\\\"\\n\")\" failed at t/unit-tests/t-basic.c:61\n+\t# check \"!strcmp(\"\\thello\\\\\\\\\", \"there\\\"\\n\")\" failed at t/helper/test-example-tap.c:62\n \t# left: \"\\011hello\\\\\\\\\"\n \t# right: \"there\\\"\\012\"\n-\t# check \"!strcmp(\"NULL\", NULL)\" failed at t/unit-tests/t-basic.c:62\n+\t# check \"!strcmp(\"NULL\", NULL)\" failed at t/helper/test-example-tap.c:63\n \t# left: \"NULL\"\n \t# right: NULL\n-\t# check \"${SQ}a${SQ} == ${SQ}\\n${SQ}\" failed at t/unit-tests/t-basic.c:63\n+\t# check \"${SQ}a${SQ} == ${SQ}\\n${SQ}\" failed at t/helper/test-example-tap.c:64\n \t# left: ${SQ}a${SQ}\n \t# right: ${SQ}\\012${SQ}\n-\t# check \"${SQ}\\\\\\\\${SQ} == ${SQ}\\\\${SQ}${SQ}\" failed at t/unit-tests/t-basic.c:64\n+\t# check \"${SQ}\\\\\\\\${SQ} == ${SQ}\\\\${SQ}${SQ}\" failed at t/helper/test-example-tap.c:65\n \t# left: ${SQ}\\\\\\\\${SQ}\n \t# right: ${SQ}\\\\${SQ}${SQ}\n \tnot ok 17 - messages from failing string and char comparison\n-\t# BUG: test has no checks at t/unit-tests/t-basic.c:91\n+\t# BUG: test has no checks at t/helper/test-example-tap.c:92\n \tnot ok 18 - test with no checks\n \tok 19 - test with no checks returns 0\n \t1..19\n \tEOF\n \n-\t! \"$GIT_BUILD_DIR\"/t/unit-tests/bin/t-basic >actual &&\n+\t! test-tool example-tap >actual &&\n \ttest_cmp expect actual\n '\n \n-- \n2.44.0.769.g3c40516874-goog\n\n"},{"id":"493429","messageId":"24f47f8fc7e931c2e15fa985635e3036ec1123ba.1713985716.git.steadmon@google.com","threadId":"60754","inReplyTo":"cover.1713985716.git.steadmon@google.com","subject":"[PATCH v4 2/7] test-tool run-command testsuite: get shell from env","fromName":"Josh Steadmon","fromEmail":"steadmon@google.com","sentAt":"2024-04-24T19:14:08Z","receivedAt":"2024-04-24T19:14:20Z","isPatch":true,"sender":{"key":"steadmon@google.com","avatar":"https://avatars.githubusercontent.com/u/2654920?v=4"},"body":"When running tests through `test-tool run-command testsuite`, we\ncurrently hardcode `sh` as the command interpreter. As discussed in [1],\nthis is incorrect, and we should be using the shell set in\nTEST_SHELL_PATH instead.\n\nAdd a shell_path field in struct testsuite so that we can pass this to\nthe task runner callback. If this is non-null, we'll use it as the\nargv[0] of the subprocess. Otherwise, we'll just execute the test\nprogram directly. We will use this feature in a later commit to enable\nrunning binary executable unit tests.\n\nHowever, for now when setting up the struct testsuite in testsuite(),\nuse the value of TEST_SHELL_PATH if it's set, otherwise keep the\noriginal behavior by defaulting to `sh`.\n\n[1] https://lore.kernel.org/git/20240123005913.GB835964@coredump.intra.peff.net/\n\nSigned-off-by: Josh Steadmon \n---\n t/helper/test-run-command.c | 9 ++++++++-\n 1 file changed, 8 insertions(+), 1 deletion(-)\n\ndiff --git a/t/helper/test-run-command.c b/t/helper/test-run-command.c\nindex c0ed8722c8..a41a54d9cb 100644\n--- a/t/helper/test-run-command.c\n+++ b/t/helper/test-run-command.c\n@@ -65,6 +65,7 @@ struct testsuite {\n \tstruct string_list tests, failed;\n \tint next;\n \tint quiet, immediate, verbose, verbose_log, trace, write_junit_xml;\n+\tconst char *shell_path;\n };\n #define TESTSUITE_INIT { \\\n \t.tests = STRING_LIST_INIT_DUP, \\\n@@ -80,7 +81,9 @@ static int next_test(struct child_process *cp, struct strbuf *err, void *cb,\n \t\treturn 0;\n \n \ttest = suite->tests.items[suite->next++].string;\n-\tstrvec_pushl(&cp->args, \"sh\", test, NULL);\n+\tif (suite->shell_path)\n+\t\tstrvec_push(&cp->args, suite->shell_path);\n+\tstrvec_push(&cp->args, test);\n \tif (suite->quiet)\n \t\tstrvec_push(&cp->args, \"--quiet\");\n \tif (suite->immediate)\n@@ -162,6 +165,10 @@ static int testsuite(int argc, const char **argv)\n \tif (max_jobs <= 0)\n \t\tmax_jobs = online_cpus();\n \n+\tsuite.shell_path = getenv(\"TEST_SHELL_PATH\");\n+\tif (!suite.shell_path)\n+\t\tsuite.shell_path = \"sh\";\n+\n \tdir = opendir(\".\");\n \tif (!dir)\n \t\tdie(\"Could not open the current directory\");\n-- \n2.44.0.769.g3c40516874-goog\n\n"},{"id":"493430","messageId":"4a16a3ec241d83b3de424ce835db24eee6e46f0f.1713985716.git.steadmon@google.com","threadId":"60754","inReplyTo":"cover.1713985716.git.steadmon@google.com","subject":"[PATCH v4 3/7] test-tool run-command testsuite: remove hardcoded filter","fromName":"Josh Steadmon","fromEmail":"steadmon@google.com","sentAt":"2024-04-24T19:14:09Z","receivedAt":"2024-04-24T19:14:22Z","isPatch":true,"sender":{"key":"steadmon@google.com","avatar":"https://avatars.githubusercontent.com/u/2654920?v=4"},"body":"`test-tool run-command testsuite` currently assumes that it will only be\nrunning the shell test suite, and therefore filters out anything that\ndoes not match a hardcoded pattern of \"t[0-9][0-9][0-9][0-9]-*.sh\".\n\nLater in this series, we'll adapt `test-tool run-command testsuite` to\nalso support unit tests, which do not follow the same naming conventions\nas the shell tests, so this hardcoded pattern is inconvenient.\n\nSince `testsuite` also allows specifying patterns on the command-line,\nlet's just remove this pattern. As noted in [1], there are no longer any\nuses of `testsuite` in our codebase, it should be OK to break backwards\ncompatibility in this case. We also add a new filter to avoid trying to\nexecute \".\" and \"..\", so that users who wish to execute every test in a\ndirectory can do so without specifying a pattern.\n\n[1] https://lore.kernel.org/git/850ea42c-f103-68d5-896b-9120e2628686@gmx.de/\n\nSigned-off-by: Josh Steadmon \n---\n t/helper/test-run-command.c | 4 +---\n 1 file changed, 1 insertion(+), 3 deletions(-)\n\ndiff --git a/t/helper/test-run-command.c b/t/helper/test-run-command.c\nindex a41a54d9cb..e6bd792274 100644\n--- a/t/helper/test-run-command.c\n+++ b/t/helper/test-run-command.c\n@@ -175,9 +175,7 @@ static int testsuite(int argc, const char **argv)\n \twhile ((d = readdir(dir))) {\n \t\tconst char *p = d->d_name;\n \n-\t\tif (*p != 't' || !isdigit(p[1]) || !isdigit(p[2]) ||\n-\t\t !isdigit(p[3]) || !isdigit(p[4]) || p[5] != '-' ||\n-\t\t !ends_with(p, \".sh\"))\n+\t\tif (!strcmp(p, \".\") || !strcmp(p, \"..\"))\n \t\t\tcontinue;\n \n \t\t/* No pattern: match all */\n-- \n2.44.0.769.g3c40516874-goog\n\n"},{"id":"493431","messageId":"abc9a7afe8c1ab92bfbd2b9eea672bdd9ff3c107.1713985716.git.steadmon@google.com","threadId":"60754","inReplyTo":"cover.1713985716.git.steadmon@google.com","subject":"[PATCH v4 4/7] test-tool run-command testsuite: support unit tests","fromName":"Josh Steadmon","fromEmail":"steadmon@google.com","sentAt":"2024-04-24T19:14:10Z","receivedAt":"2024-04-24T19:14:24Z","isPatch":true,"sender":{"key":"steadmon@google.com","avatar":"https://avatars.githubusercontent.com/u/2654920?v=4"},"body":"Teach the testsuite runner in `test-tool run-command testsuite` how to\nrun unit tests: if TEST_SHELL_PATH is not set, run the programs directly\nfrom CWD, rather than defaulting to \"sh\" as an interpreter.\n\nWith this change, you can now use test-tool to run the unit tests:\n$ make\n$ cd t/unit-tests/bin\n$ ../../helper/test-tool run-command testsuite\n\nThis should be helpful on Windows to allow running tests without\nrequiring Perl (for `prove`), as discussed in [1] and [2].\n\nThis again breaks backwards compatibility, as it is now required to set\nTEST_SHELL_PATH properly for executing shell scripts, but again, as\nnoted in [2], there are no longer any such invocations in our codebase.\n\n[1] https://lore.kernel.org/git/nycvar.QRO.7.76.6.2109091323150.59@tvgsbejvaqbjf.bet/\n[2] https://lore.kernel.org/git/850ea42c-f103-68d5-896b-9120e2628686@gmx.de/\n\nSigned-off-by: Josh Steadmon \n---\n t/helper/test-run-command.c | 17 ++++++++++++++---\n 1 file changed, 14 insertions(+), 3 deletions(-)\n\ndiff --git a/t/helper/test-run-command.c b/t/helper/test-run-command.c\nindex e6bd792274..61eb1175fe 100644\n--- a/t/helper/test-run-command.c\n+++ b/t/helper/test-run-command.c\n@@ -158,6 +158,8 @@ static int testsuite(int argc, const char **argv)\n \t\t.task_finished = test_finished,\n \t\t.data = &suite,\n \t};\n+\tstruct strbuf progpath = STRBUF_INIT;\n+\tsize_t path_prefix_len;\n \n \targc = parse_options(argc, argv, NULL, options,\n \t\t\ttestsuite_usage, PARSE_OPT_STOP_AT_NON_OPTION);\n@@ -165,9 +167,13 @@ static int testsuite(int argc, const char **argv)\n \tif (max_jobs <= 0)\n \t\tmax_jobs = online_cpus();\n \n+\t/*\n+\t * If we run without a shell, execute the programs directly from CWD.\n+\t */\n \tsuite.shell_path = getenv(\"TEST_SHELL_PATH\");\n \tif (!suite.shell_path)\n-\t\tsuite.shell_path = \"sh\";\n+\t\tstrbuf_addstr(&progpath, \"./\");\n+\tpath_prefix_len = progpath.len;\n \n \tdir = opendir(\".\");\n \tif (!dir)\n@@ -180,13 +186,17 @@ static int testsuite(int argc, const char **argv)\n \n \t\t/* No pattern: match all */\n \t\tif (!argc) {\n-\t\t\tstring_list_append(&suite.tests, p);\n+\t\t\tstrbuf_setlen(&progpath, path_prefix_len);\n+\t\t\tstrbuf_addstr(&progpath, p);\n+\t\t\tstring_list_append(&suite.tests, progpath.buf);\n \t\t\tcontinue;\n \t\t}\n \n \t\tfor (i = 0; i < argc; i++)\n \t\t\tif (!wildmatch(argv[i], p, 0)) {\n-\t\t\t\tstring_list_append(&suite.tests, p);\n+\t\t\t\tstrbuf_setlen(&progpath, path_prefix_len);\n+\t\t\t\tstrbuf_addstr(&progpath, p);\n+\t\t\t\tstring_list_append(&suite.tests, progpath.buf);\n \t\t\t\tbreak;\n \t\t\t}\n \t}\n@@ -213,6 +223,7 @@ static int testsuite(int argc, const char **argv)\n \n \tstring_list_clear(&suite.tests, 0);\n \tstring_list_clear(&suite.failed, 0);\n+\tstrbuf_release(&progpath);\n \n \treturn ret;\n }\n-- \n2.44.0.769.g3c40516874-goog\n\n"},{"id":"493432","messageId":"a8bbff2c6bcea6a8874d2a145cffe370d6cbddfb.1713985716.git.steadmon@google.com","threadId":"60754","inReplyTo":"cover.1713985716.git.steadmon@google.com","subject":"[PATCH v4 5/7] unit tests: add rule for running with test-tool","fromName":"Josh Steadmon","fromEmail":"steadmon@google.com","sentAt":"2024-04-24T19:14:11Z","receivedAt":"2024-04-24T19:14:26Z","isPatch":true,"sender":{"key":"steadmon@google.com","avatar":"https://avatars.githubusercontent.com/u/2654920?v=4"},"body":"In the previous commit, we added support in test-tool for running\ncollections of unit tests. Now, add rules in t/Makefile for running in\nthis way.\n\nThis new rule can be executed from the top-level Makefile via\n`make DEFAULT_UNIT_TEST_TARGET=unit-tests-test-tool unit-tests`, or by\nsetting DEFAULT_UNIT_TEST_TARGET in config.mak.\n\nSigned-off-by: Josh Steadmon \n---\n Makefile | 2 +-\n t/Makefile | 10 +++++++++-\n 2 files changed, 10 insertions(+), 2 deletions(-)\n\ndiff --git a/Makefile b/Makefile\nindex ba55d817ee..b0d1f04b4d 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -3870,5 +3870,5 @@ $(UNIT_TEST_PROGS): $(UNIT_TEST_BIN)/%$X: $(UNIT_TEST_DIR)/%.o $(UNIT_TEST_DIR)/\n \n .PHONY: build-unit-tests unit-tests\n build-unit-tests: $(UNIT_TEST_PROGS)\n-unit-tests: $(UNIT_TEST_PROGS)\n+unit-tests: $(UNIT_TEST_PROGS) t/helper/test-tool$X\n \t$(MAKE) -C t/ unit-tests\ndiff --git a/t/Makefile b/t/Makefile\nindex 4861edafe6..0ae04f1e42 100644\n--- a/t/Makefile\n+++ b/t/Makefile\n@@ -49,6 +49,7 @@ CHAINLINT = '$(PERL_PATH_SQ)' chainlint.pl\n UNIT_TEST_SOURCES = $(wildcard unit-tests/t-*.c)\n UNIT_TEST_PROGRAMS = $(patsubst unit-tests/%.c,unit-tests/bin/%$(X),$(UNIT_TEST_SOURCES))\n UNIT_TESTS = $(sort $(UNIT_TEST_PROGRAMS))\n+UNIT_TESTS_NO_DIR = $(notdir $(UNIT_TESTS))\n \n # `test-chainlint` (which is a dependency of `test-lint`, `test` and `prove`)\n # checks all tests in all scripts via a single invocation, so tell individual\n@@ -76,7 +77,7 @@ $(T):\n $(UNIT_TESTS):\n \t@echo \"*** $@ ***\"; $@\n \n-.PHONY: unit-tests unit-tests-raw unit-tests-prove\n+.PHONY: unit-tests unit-tests-raw unit-tests-prove unit-tests-test-tool\n unit-tests: $(DEFAULT_UNIT_TEST_TARGET)\n \n unit-tests-raw: $(UNIT_TESTS)\n@@ -84,6 +85,13 @@ unit-tests-raw: $(UNIT_TESTS)\n unit-tests-prove:\n \t@echo \"*** prove - unit tests ***\"; $(PROVE) $(GIT_PROVE_OPTS) $(UNIT_TESTS)\n \n+unit-tests-test-tool:\n+\t@echo \"*** test-tool - unit tests **\"\n+\t( \\\n+\t\tcd unit-tests/bin && \\\n+\t\t../../helper/test-tool$X run-command testsuite $(UNIT_TESTS_NO_DIR)\\\n+\t)\n+\n pre-clean:\n \t$(RM) -r '$(TEST_RESULTS_DIRECTORY_SQ)'\n \n-- \n2.44.0.769.g3c40516874-goog\n\n"},{"id":"493433","messageId":"0e32de1afe9cbab02c5d3476a0fc2a1ba0151dcf.1713985716.git.steadmon@google.com","threadId":"60754","inReplyTo":"cover.1713985716.git.steadmon@google.com","subject":"[PATCH v4 6/7] t/Makefile: run unit tests alongside shell tests","fromName":"Josh Steadmon","fromEmail":"steadmon@google.com","sentAt":"2024-04-24T19:14:12Z","receivedAt":"2024-04-24T19:14:27Z","isPatch":true,"sender":{"key":"steadmon@google.com","avatar":"https://avatars.githubusercontent.com/u/2654920?v=4"},"body":"From: Jeff King \n\nAdd a wrapper script to allow `prove` to run both shell tests and unit\ntests from a single invocation. This avoids issues around running prove\ntwice in CI, as discussed in [1].\n\nAdditionally, this moves the unit tests into the main dev workflow, so\nthat errors can be spotted more quickly. Accordingly, we remove the\nseparate unit tests step for Linux CI. (We leave the Windows CI\nunit-test step as-is, because the sharding scheme there involves\nselecting specific test files rather than running `make test`.)\n\n[1] https://lore.kernel.org/git/pull.1613.git.1699894837844.gitgitgadget@gmail.com/\n\nSigned-off-by: Jeff King \nSigned-off-by: Josh Steadmon \n---\n ci/run-build-and-tests.sh | 2 --\n t/Makefile | 2 +-\n t/run-test.sh | 17 +++++++++++++++++\n 3 files changed, 18 insertions(+), 3 deletions(-)\n create mode 100755 t/run-test.sh\n\ndiff --git a/ci/run-build-and-tests.sh b/ci/run-build-and-tests.sh\nindex 7a1466b868..2528f25e31 100755\n--- a/ci/run-build-and-tests.sh\n+++ b/ci/run-build-and-tests.sh\n@@ -50,8 +50,6 @@ if test -n \"$run_tests\"\n then\n \tgroup \"Run tests\" make test ||\n \thandle_failed_tests\n-\tgroup \"Run unit tests\" \\\n-\t\tmake DEFAULT_UNIT_TEST_TARGET=unit-tests-prove unit-tests\n fi\n check_unignored_build_artifacts\n \ndiff --git a/t/Makefile b/t/Makefile\nindex 0ae04f1e42..b2eb9f770b 100644\n--- a/t/Makefile\n+++ b/t/Makefile\n@@ -68,7 +68,7 @@ failed:\n \ttest -z \"$$failed\" || $(MAKE) $$failed\n \n prove: pre-clean check-chainlint $(TEST_LINT)\n-\t@echo \"*** prove ***\"; $(CHAINLINTSUPPRESS) $(PROVE) --exec '$(TEST_SHELL_PATH_SQ)' $(GIT_PROVE_OPTS) $(T) :: $(GIT_TEST_OPTS)\n+\t@echo \"*** prove (shell & unit tests) ***\"; $(CHAINLINTSUPPRESS) TEST_SHELL_PATH='$(TEST_SHELL_PATH_SQ)' $(PROVE) --exec ./run-test.sh $(GIT_PROVE_OPTS) $(T) $(UNIT_TESTS) :: $(GIT_TEST_OPTS)\n \t$(MAKE) clean-except-prove-cache\n \n $(T):\ndiff --git a/t/run-test.sh b/t/run-test.sh\nnew file mode 100755\nindex 0000000000..a0e2dce5e0\n--- /dev/null\n+++ b/t/run-test.sh\n@@ -0,0 +1,17 @@\n+#!/bin/sh\n+\n+# A simple wrapper to run shell tests via TEST_SHELL_PATH,\n+# or exec unit tests directly.\n+\n+case \"$1\" in\n+*.sh)\n+\tif test -z \"${TEST_SHELL_PATH+set}\" ; then\n+\t\techo \"ERROR: TEST_SHELL_PATH is not set\" >&2\n+\t\texit 1\n+\tfi\n+\texec ${TEST_SHELL_PATH} \"$@\"\n+\t;;\n+*)\n+\texec \"$@\"\n+\t;;\n+esac\n-- \n2.44.0.769.g3c40516874-goog\n\n"},{"id":"493434","messageId":"c562515293d883296b1732fa1691369ac03e3213.1713985716.git.steadmon@google.com","threadId":"60754","inReplyTo":"cover.1713985716.git.steadmon@google.com","subject":"[PATCH v4 7/7] ci: use test-tool as unit test runner on Windows","fromName":"Josh Steadmon","fromEmail":"steadmon@google.com","sentAt":"2024-04-24T19:14:13Z","receivedAt":"2024-04-24T19:14:29Z","isPatch":true,"sender":{"key":"steadmon@google.com","avatar":"https://avatars.githubusercontent.com/u/2654920?v=4"},"body":"Although the previous commit changed t/Makefile to run unit tests\nalongside shell tests, the Windows CI still needs a separate unit-tests\nstep due to how the test sharding works.\n\nWe want to avoid using `prove` as a test running on Windows due to\nperformance issues [1], so use the new test-tool runner instead.\n\n[1] https://lore.kernel.org/git/850ea42c-f103-68d5-896b-9120e2628686@gmx.de/\n\nSigned-off-by: Josh Steadmon \n---\n ci/run-test-slice.sh | 2 +-\n 1 file changed, 1 insertion(+), 1 deletion(-)\n\ndiff --git a/ci/run-test-slice.sh b/ci/run-test-slice.sh\nindex ae8094382f..e167e646f7 100755\n--- a/ci/run-test-slice.sh\n+++ b/ci/run-test-slice.sh\n@@ -17,7 +17,7 @@ handle_failed_tests\n \n # We only have one unit test at the moment, so run it in the first slice\n if [ \"$1\" == \"0\" ] ; then\n-\tgroup \"Run unit tests\" make --quiet -C t unit-tests-prove\n+\tgroup \"Run unit tests\" make --quiet -C t unit-tests-test-tool\n fi\n \n check_unignored_build_artifacts\n-- \n2.44.0.769.g3c40516874-goog\n\n"},{"id":"493440","messageId":"xmqqo79yxxk7.fsf@gitster.g","threadId":"60754","inReplyTo":"0e32de1afe9cbab02c5d3476a0fc2a1ba0151dcf.1713985716.git.steadmon@google.com","subject":"Re: [PATCH v4 6/7] t/Makefile: run unit tests alongside shell tests","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2024-04-24T21:25:44Z","receivedAt":"2024-04-24T21:25:50Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Josh Steadmon writes:\n\n> +case \"$1\" in\n> +*.sh)\n> +\tif test -z \"${TEST_SHELL_PATH+set}\" ; then\n> +\t\techo \"ERROR: TEST_SHELL_PATH is not set\" >&2\n\nStyle.\n\nAs an empty string is not a reasonable value for this variable (and\nyou do not quote ${TEST_SHELL_PATH} when you use it in \"exec\" below),\n\n\tif test -z \"${TEST_SHELL_PATH:+set}\"\n\tthen\n\t\techo >&2 \"ERROR: TEST_SHELL_PATH is not set or empty\"\n\nmay be what we want here.\n\n> +\t\texit 1\n> +\tfi\n> +\texec ${TEST_SHELL_PATH} \"$@\"\n> +\t;;\n> +*)\n> +\texec \"$@\"\n> +\t;;\n> +esac\n\nOther than that, the update in this iteration looks reasonable to\nme.\n\nThanks.\n"},{"id":"493807","messageId":"vspumeff4mdkxjnbqgrkwh25v5wh45zpq7yvrbkfagaknzrv54@m3mz4rtunn6w","threadId":"60754","inReplyTo":"xmqqo79yxxk7.fsf@gitster.g","subject":"Re: [PATCH v4 6/7] t/Makefile: run unit tests alongside shell tests","fromName":"Josh Steadmon","fromEmail":"steadmon@google.com","sentAt":"2024-04-30T19:49:19Z","receivedAt":"2024-04-30T19:49:26Z","isPatch":true,"sender":{"key":"steadmon@google.com","avatar":"https://avatars.githubusercontent.com/u/2654920?v=4"},"body":"On 2024.04.24 14:25, Junio C Hamano wrote:\n> Josh Steadmon writes:\n> \n> > +case \"$1\" in\n> > +*.sh)\n> > +\tif test -z \"${TEST_SHELL_PATH+set}\" ; then\n> > +\t\techo \"ERROR: TEST_SHELL_PATH is not set\" >&2\n> \n> Style.\n> \n> As an empty string is not a reasonable value for this variable (and\n> you do not quote ${TEST_SHELL_PATH} when you use it in \"exec\" below),\n> \n> \tif test -z \"${TEST_SHELL_PATH:+set}\"\n> \tthen\n> \t\techo >&2 \"ERROR: TEST_SHELL_PATH is not set or empty\"\n> \n> may be what we want here.\n> \n> > +\t\texit 1\n> > +\tfi\n> > +\texec ${TEST_SHELL_PATH} \"$@\"\n> > +\t;;\n> > +*)\n> > +\texec \"$@\"\n> > +\t;;\n> > +esac\n> \n> Other than that, the update in this iteration looks reasonable to\n> me.\n> \n> Thanks.\n\nFixed in V5, thanks.\n"},{"id":"493808","messageId":"cover.1714506612.git.steadmon@google.com","threadId":"60754","inReplyTo":"cover.1705443632.git.steadmon@google.com","subject":"[PATCH v5 0/7] test-tool: add unit test suite runner","fromName":"Josh Steadmon","fromEmail":"steadmon@google.com","sentAt":"2024-04-30T19:55:50Z","receivedAt":"2024-04-30T19:56:00Z","isPatch":true,"sender":{"key":"steadmon@google.com","avatar":"https://avatars.githubusercontent.com/u/2654920?v=4"},"body":"For various reasons (see discussion at [1]) we would like an alternative\nto `prove` for running test suites (including the unit tests) on\nWindows.\n\n[1] https://lore.kernel.org/git/pull.1613.git.1699894837844.gitgitgadget@gmail.com/\n\nThis series extends the existing `test-tool run-command testsuite` to\nsupport running unit tests. In addition, it includes some small\ncleanups:\n* move t-basic out of the unit-tests directory\n* don't hardcode the shell for running tests in `test-tool ... testsuite`\n* don't hardcode a test name filter in `test-tool ... testsuite`\n* add a test wrapper script to allow unit tests and the shell test suite\n to run in a single `prove` process\n\nChanges in V5:\n* Style fix in t/run-test.sh\n\nChanges in V4:\n* Set TEST_SHELL_PATH when running the `prove` test target, and error\n out of run-test.sh if TEST_SHELL_PATH is not set.\n\nChanges in V3:\n* Added new patch (#7) to use the new test-tool runner for unit tests in\n Windows CI.\n* Restored the explicit sort call in t/Makefile, for backwards\n compatibility with older GNU Make versions.\n* Rebased the series on the latest jk/unit-tests-buildfix branch.\n* Fixed header include order in patch #1.\n* Removed a paragraph in patch #1's commit message that is obsolete now\n that we're building the list of test files from the sources rather\n than by globbing.\n* Added a note in patch #2 that setting a NULL suite.shell_path will be\n used in a later commit.\n* Clarified up some sloppy wording in commit messages and comments in\n t/helper/test-run-command.c.\n\nChanges in V2:\n* Rebased the series on the latest jk/unit-tests-buildfix branch.\n* Patch 1: move t-basic to a test-tool subcommand rather than a new\n executable under t/t0080/\n* New patch 2: get the shell path from TEST_SHELL_PATH in\n `test-tool run-command testsuite`\n* New patch 3: remove the hardcoded filename filter in\n `test-tool run-command testsuite`\n* Patch 4 (previously 2): simplified now that we no longer need to add\n any command-line flags to support unit tests\n* Patch 5 (previously 3): avoid trying to run cmake *.pdb files by using\n the unit test list built in the makefile in jk/unit-tests-buildfix.\n\n\nJeff King (1):\n t/Makefile: run unit tests alongside shell tests\n\nJosh Steadmon (6):\n t0080: turn t-basic unit test into a helper\n test-tool run-command testsuite: get shell from env\n test-tool run-command testsuite: remove hardcoded filter\n test-tool run-command testsuite: support unit tests\n unit tests: add rule for running with test-tool\n ci: use test-tool as unit test runner on Windows\n\n Makefile | 6 ++--\n ci/run-build-and-tests.sh | 2 --\n ci/run-test-slice.sh | 2 +-\n t/Makefile | 14 ++++++++--\n .../t-basic.c => helper/test-example-tap.c} | 5 ++--\n t/helper/test-run-command.c | 28 +++++++++++++++----\n t/helper/test-tool.c | 1 +\n t/helper/test-tool.h | 1 +\n t/run-test.sh | 18 ++++++++++++\n t/t0080-unit-test-output.sh | 24 ++++++++--------\n 10 files changed, 72 insertions(+), 29 deletions(-)\n rename t/{unit-tests/t-basic.c => helper/test-example-tap.c} (95%)\n create mode 100755 t/run-test.sh\n\nRange-diff against v4:\n1: 6777451100 = 1: 6777451100 t0080: turn t-basic unit test into a helper\n2: 24f47f8fc7 = 2: 24f47f8fc7 test-tool run-command testsuite: get shell from env\n3: 4a16a3ec24 = 3: 4a16a3ec24 test-tool run-command testsuite: remove hardcoded filter\n4: abc9a7afe8 = 4: abc9a7afe8 test-tool run-command testsuite: support unit tests\n5: a8bbff2c6b = 5: a8bbff2c6b unit tests: add rule for running with test-tool\n6: 0e32de1afe ! 6: c6606446c4 t/Makefile: run unit tests alongside shell tests\n @@ t/run-test.sh (new)\n +\n +case \"$1\" in\n +*.sh)\n -+ if test -z \"${TEST_SHELL_PATH+set}\" ; then\n -+ echo \"ERROR: TEST_SHELL_PATH is not set\" >&2\n ++ if test -z \"${TEST_SHELL_PATH+set}\"\n ++ then\n ++ echo >&2 \"ERROR: TEST_SHELL_PATH is empty or not set\"\n + exit 1\n + fi\n + exec ${TEST_SHELL_PATH} \"$@\"\n7: c562515293 = 7: 4a92131ab9 ci: use test-tool as unit test runner on Windows\n\nbase-commit: 4904a4d08cc085716df12ce713ae7ee3d5ecb75a\n-- \n2.45.0.rc0.197.gbae5840b3b-goog\n\n"},{"id":"493809","messageId":"677745110078501e611faf3d1facd1518e0453a2.1714506612.git.steadmon@google.com","threadId":"60754","inReplyTo":"cover.1714506612.git.steadmon@google.com","subject":"[PATCH v5 1/7] t0080: turn t-basic unit test into a helper","fromName":"Josh Steadmon","fromEmail":"steadmon@google.com","sentAt":"2024-04-30T19:55:51Z","receivedAt":"2024-04-30T19:56:02Z","isPatch":true,"sender":{"key":"steadmon@google.com","avatar":"https://avatars.githubusercontent.com/u/2654920?v=4"},"body":"While t/unit-tests/t-basic.c uses the unit-test framework added in\ne137fe3b29 (unit tests: add TAP unit test framework, 2023-11-09), it is\nnot a true unit test in that it intentionally fails in order to exercise\nvarious codepaths in the unit-test framework. Thus, we intentionally\nexclude it when running unit tests through the various t/Makefile\ntargets. Instead, it is executed by t0080-unit-test-output.sh, which\nverifies its output follows the TAP format expected for the various\npass, skip, or fail cases.\n\nAs such, it makes more sense for t-basic to be a helper item for\nt0080-unit-test-output.sh, so let's move it to\nt/helper/test-example-tap.c and adjust Makefiles as necessary.\n\nSigned-off-by: Josh Steadmon \n---\n Makefile | 4 ++--\n t/Makefile | 2 +-\n .../t-basic.c => helper/test-example-tap.c} | 5 ++--\n t/helper/test-tool.c | 1 +\n t/helper/test-tool.h | 1 +\n t/t0080-unit-test-output.sh | 24 +++++++++----------\n 6 files changed, 20 insertions(+), 17 deletions(-)\n rename t/{unit-tests/t-basic.c => helper/test-example-tap.c} (95%)\n\ndiff --git a/Makefile b/Makefile\nindex 23723367b8..ba55d817ee 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -802,6 +802,7 @@ TEST_BUILTINS_OBJS += test-dump-split-index.o\n TEST_BUILTINS_OBJS += test-dump-untracked-cache.o\n TEST_BUILTINS_OBJS += test-env-helper.o\n TEST_BUILTINS_OBJS += test-example-decorate.o\n+TEST_BUILTINS_OBJS += test-example-tap.o\n TEST_BUILTINS_OBJS += test-find-pack.o\n TEST_BUILTINS_OBJS += test-fsmonitor-client.o\n TEST_BUILTINS_OBJS += test-genrandom.o\n@@ -1338,7 +1339,6 @@ THIRD_PARTY_SOURCES += compat/regex/%\n THIRD_PARTY_SOURCES += sha1collisiondetection/%\n THIRD_PARTY_SOURCES += sha1dc/%\n \n-UNIT_TEST_PROGRAMS += t-basic\n UNIT_TEST_PROGRAMS += t-mem-pool\n UNIT_TEST_PROGRAMS += t-strbuf\n UNIT_TEST_PROGRAMS += t-ctype\n@@ -3218,7 +3218,7 @@ perf: all\n \n .PRECIOUS: $(TEST_OBJS)\n \n-t/helper/test-tool$X: $(patsubst %,t/helper/%,$(TEST_BUILTINS_OBJS))\n+t/helper/test-tool$X: $(patsubst %,t/helper/%,$(TEST_BUILTINS_OBJS)) $(UNIT_TEST_DIR)/test-lib.o\n \n t/helper/test-%$X: t/helper/test-%.o GIT-LDFLAGS $(GITLIBS) $(REFTABLE_TEST_LIB)\n \t$(QUIET_LINK)$(CC) $(ALL_CFLAGS) -o $@ $(ALL_LDFLAGS) $(filter %.o,$^) $(filter %.a,$^) $(LIBS)\ndiff --git a/t/Makefile b/t/Makefile\nindex 2d95046f26..4861edafe6 100644\n--- a/t/Makefile\n+++ b/t/Makefile\n@@ -48,7 +48,7 @@ CHAINLINTTESTS = $(sort $(patsubst chainlint/%.test,%,$(wildcard chainlint/*.tes\n CHAINLINT = '$(PERL_PATH_SQ)' chainlint.pl\n UNIT_TEST_SOURCES = $(wildcard unit-tests/t-*.c)\n UNIT_TEST_PROGRAMS = $(patsubst unit-tests/%.c,unit-tests/bin/%$(X),$(UNIT_TEST_SOURCES))\n-UNIT_TESTS = $(sort $(filter-out unit-tests/bin/t-basic%,$(UNIT_TEST_PROGRAMS)))\n+UNIT_TESTS = $(sort $(UNIT_TEST_PROGRAMS))\n \n # `test-chainlint` (which is a dependency of `test-lint`, `test` and `prove`)\n # checks all tests in all scripts via a single invocation, so tell individual\ndiff --git a/t/unit-tests/t-basic.c b/t/helper/test-example-tap.c\nsimilarity index 95%\nrename from t/unit-tests/t-basic.c\nrename to t/helper/test-example-tap.c\nindex fda1ae59a6..d072ad559f 100644\n--- a/t/unit-tests/t-basic.c\n+++ b/t/helper/test-example-tap.c\n@@ -1,4 +1,5 @@\n-#include \"test-lib.h\"\n+#include \"test-tool.h\"\n+#include \"t/unit-tests/test-lib.h\"\n \n /*\n * The purpose of this \"unit test\" is to verify a few invariants of the unit\n@@ -69,7 +70,7 @@ static void t_empty(void)\n \t; /* empty */\n }\n \n-int cmd_main(int argc, const char **argv)\n+int cmd__example_tap(int argc, const char **argv)\n {\n \ttest_res = TEST(check_res = check_int(1, ==, 1), \"passing test\");\n \tTEST(t_res(1), \"passing test and assertion return 1\");\ndiff --git a/t/helper/test-tool.c b/t/helper/test-tool.c\nindex 33b9501c21..bb5c04c9c0 100644\n--- a/t/helper/test-tool.c\n+++ b/t/helper/test-tool.c\n@@ -29,6 +29,7 @@ static struct test_cmd cmds[] = {\n \t{ \"dump-untracked-cache\", cmd__dump_untracked_cache },\n \t{ \"env-helper\", cmd__env_helper },\n \t{ \"example-decorate\", cmd__example_decorate },\n+\t{ \"example-tap\", cmd__example_tap },\n \t{ \"find-pack\", cmd__find_pack },\n \t{ \"fsmonitor-client\", cmd__fsmonitor_client },\n \t{ \"genrandom\", cmd__genrandom },\ndiff --git a/t/helper/test-tool.h b/t/helper/test-tool.h\nindex b72f07ded9..38001bd1c6 100644\n--- a/t/helper/test-tool.h\n+++ b/t/helper/test-tool.h\n@@ -23,6 +23,7 @@ int cmd__dump_untracked_cache(int argc, const char **argv);\n int cmd__dump_reftable(int argc, const char **argv);\n int cmd__env_helper(int argc, const char **argv);\n int cmd__example_decorate(int argc, const char **argv);\n+int cmd__example_tap(int argc, const char **argv);\n int cmd__find_pack(int argc, const char **argv);\n int cmd__fsmonitor_client(int argc, const char **argv);\n int cmd__genrandom(int argc, const char **argv);\ndiff --git a/t/t0080-unit-test-output.sh b/t/t0080-unit-test-output.sh\nindex 961b54b06c..83b1e3b7f5 100755\n--- a/t/t0080-unit-test-output.sh\n+++ b/t/t0080-unit-test-output.sh\n@@ -8,50 +8,50 @@ test_expect_success 'TAP output from unit tests' '\n \tcat >expect <<-EOF &&\n \tok 1 - passing test\n \tok 2 - passing test and assertion return 1\n-\t# check \"1 == 2\" failed at t/unit-tests/t-basic.c:76\n+\t# check \"1 == 2\" failed at t/helper/test-example-tap.c:77\n \t# left: 1\n \t# right: 2\n \tnot ok 3 - failing test\n \tok 4 - failing test and assertion return 0\n \tnot ok 5 - passing TEST_TODO() # TODO\n \tok 6 - passing TEST_TODO() returns 1\n-\t# todo check ${SQ}check(x)${SQ} succeeded at t/unit-tests/t-basic.c:25\n+\t# todo check ${SQ}check(x)${SQ} succeeded at t/helper/test-example-tap.c:26\n \tnot ok 7 - failing TEST_TODO()\n \tok 8 - failing TEST_TODO() returns 0\n-\t# check \"0\" failed at t/unit-tests/t-basic.c:30\n+\t# check \"0\" failed at t/helper/test-example-tap.c:31\n \t# skipping test - missing prerequisite\n-\t# skipping check ${SQ}1${SQ} at t/unit-tests/t-basic.c:32\n+\t# skipping check ${SQ}1${SQ} at t/helper/test-example-tap.c:33\n \tok 9 - test_skip() # SKIP\n \tok 10 - skipped test returns 1\n \t# skipping test - missing prerequisite\n \tok 11 - test_skip() inside TEST_TODO() # SKIP\n \tok 12 - test_skip() inside TEST_TODO() returns 1\n-\t# check \"0\" failed at t/unit-tests/t-basic.c:48\n+\t# check \"0\" failed at t/helper/test-example-tap.c:49\n \tnot ok 13 - TEST_TODO() after failing check\n \tok 14 - TEST_TODO() after failing check returns 0\n-\t# check \"0\" failed at t/unit-tests/t-basic.c:56\n+\t# check \"0\" failed at t/helper/test-example-tap.c:57\n \tnot ok 15 - failing check after TEST_TODO()\n \tok 16 - failing check after TEST_TODO() returns 0\n-\t# check \"!strcmp(\"\\thello\\\\\\\\\", \"there\\\"\\n\")\" failed at t/unit-tests/t-basic.c:61\n+\t# check \"!strcmp(\"\\thello\\\\\\\\\", \"there\\\"\\n\")\" failed at t/helper/test-example-tap.c:62\n \t# left: \"\\011hello\\\\\\\\\"\n \t# right: \"there\\\"\\012\"\n-\t# check \"!strcmp(\"NULL\", NULL)\" failed at t/unit-tests/t-basic.c:62\n+\t# check \"!strcmp(\"NULL\", NULL)\" failed at t/helper/test-example-tap.c:63\n \t# left: \"NULL\"\n \t# right: NULL\n-\t# check \"${SQ}a${SQ} == ${SQ}\\n${SQ}\" failed at t/unit-tests/t-basic.c:63\n+\t# check \"${SQ}a${SQ} == ${SQ}\\n${SQ}\" failed at t/helper/test-example-tap.c:64\n \t# left: ${SQ}a${SQ}\n \t# right: ${SQ}\\012${SQ}\n-\t# check \"${SQ}\\\\\\\\${SQ} == ${SQ}\\\\${SQ}${SQ}\" failed at t/unit-tests/t-basic.c:64\n+\t# check \"${SQ}\\\\\\\\${SQ} == ${SQ}\\\\${SQ}${SQ}\" failed at t/helper/test-example-tap.c:65\n \t# left: ${SQ}\\\\\\\\${SQ}\n \t# right: ${SQ}\\\\${SQ}${SQ}\n \tnot ok 17 - messages from failing string and char comparison\n-\t# BUG: test has no checks at t/unit-tests/t-basic.c:91\n+\t# BUG: test has no checks at t/helper/test-example-tap.c:92\n \tnot ok 18 - test with no checks\n \tok 19 - test with no checks returns 0\n \t1..19\n \tEOF\n \n-\t! \"$GIT_BUILD_DIR\"/t/unit-tests/bin/t-basic >actual &&\n+\t! test-tool example-tap >actual &&\n \ttest_cmp expect actual\n '\n \n-- \n2.45.0.rc0.197.gbae5840b3b-goog\n\n"},{"id":"493810","messageId":"24f47f8fc7e931c2e15fa985635e3036ec1123ba.1714506612.git.steadmon@google.com","threadId":"60754","inReplyTo":"cover.1714506612.git.steadmon@google.com","subject":"[PATCH v5 2/7] test-tool run-command testsuite: get shell from env","fromName":"Josh Steadmon","fromEmail":"steadmon@google.com","sentAt":"2024-04-30T19:55:52Z","receivedAt":"2024-04-30T19:56:04Z","isPatch":true,"sender":{"key":"steadmon@google.com","avatar":"https://avatars.githubusercontent.com/u/2654920?v=4"},"body":"When running tests through `test-tool run-command testsuite`, we\ncurrently hardcode `sh` as the command interpreter. As discussed in [1],\nthis is incorrect, and we should be using the shell set in\nTEST_SHELL_PATH instead.\n\nAdd a shell_path field in struct testsuite so that we can pass this to\nthe task runner callback. If this is non-null, we'll use it as the\nargv[0] of the subprocess. Otherwise, we'll just execute the test\nprogram directly. We will use this feature in a later commit to enable\nrunning binary executable unit tests.\n\nHowever, for now when setting up the struct testsuite in testsuite(),\nuse the value of TEST_SHELL_PATH if it's set, otherwise keep the\noriginal behavior by defaulting to `sh`.\n\n[1] https://lore.kernel.org/git/20240123005913.GB835964@coredump.intra.peff.net/\n\nSigned-off-by: Josh Steadmon \n---\n t/helper/test-run-command.c | 9 ++++++++-\n 1 file changed, 8 insertions(+), 1 deletion(-)\n\ndiff --git a/t/helper/test-run-command.c b/t/helper/test-run-command.c\nindex c0ed8722c8..a41a54d9cb 100644\n--- a/t/helper/test-run-command.c\n+++ b/t/helper/test-run-command.c\n@@ -65,6 +65,7 @@ struct testsuite {\n \tstruct string_list tests, failed;\n \tint next;\n \tint quiet, immediate, verbose, verbose_log, trace, write_junit_xml;\n+\tconst char *shell_path;\n };\n #define TESTSUITE_INIT { \\\n \t.tests = STRING_LIST_INIT_DUP, \\\n@@ -80,7 +81,9 @@ static int next_test(struct child_process *cp, struct strbuf *err, void *cb,\n \t\treturn 0;\n \n \ttest = suite->tests.items[suite->next++].string;\n-\tstrvec_pushl(&cp->args, \"sh\", test, NULL);\n+\tif (suite->shell_path)\n+\t\tstrvec_push(&cp->args, suite->shell_path);\n+\tstrvec_push(&cp->args, test);\n \tif (suite->quiet)\n \t\tstrvec_push(&cp->args, \"--quiet\");\n \tif (suite->immediate)\n@@ -162,6 +165,10 @@ static int testsuite(int argc, const char **argv)\n \tif (max_jobs <= 0)\n \t\tmax_jobs = online_cpus();\n \n+\tsuite.shell_path = getenv(\"TEST_SHELL_PATH\");\n+\tif (!suite.shell_path)\n+\t\tsuite.shell_path = \"sh\";\n+\n \tdir = opendir(\".\");\n \tif (!dir)\n \t\tdie(\"Could not open the current directory\");\n-- \n2.45.0.rc0.197.gbae5840b3b-goog\n\n"},{"id":"493811","messageId":"4a16a3ec241d83b3de424ce835db24eee6e46f0f.1714506612.git.steadmon@google.com","threadId":"60754","inReplyTo":"cover.1714506612.git.steadmon@google.com","subject":"[PATCH v5 3/7] test-tool run-command testsuite: remove hardcoded filter","fromName":"Josh Steadmon","fromEmail":"steadmon@google.com","sentAt":"2024-04-30T19:55:53Z","receivedAt":"2024-04-30T19:56:06Z","isPatch":true,"sender":{"key":"steadmon@google.com","avatar":"https://avatars.githubusercontent.com/u/2654920?v=4"},"body":"`test-tool run-command testsuite` currently assumes that it will only be\nrunning the shell test suite, and therefore filters out anything that\ndoes not match a hardcoded pattern of \"t[0-9][0-9][0-9][0-9]-*.sh\".\n\nLater in this series, we'll adapt `test-tool run-command testsuite` to\nalso support unit tests, which do not follow the same naming conventions\nas the shell tests, so this hardcoded pattern is inconvenient.\n\nSince `testsuite` also allows specifying patterns on the command-line,\nlet's just remove this pattern. As noted in [1], there are no longer any\nuses of `testsuite` in our codebase, it should be OK to break backwards\ncompatibility in this case. We also add a new filter to avoid trying to\nexecute \".\" and \"..\", so that users who wish to execute every test in a\ndirectory can do so without specifying a pattern.\n\n[1] https://lore.kernel.org/git/850ea42c-f103-68d5-896b-9120e2628686@gmx.de/\n\nSigned-off-by: Josh Steadmon \n---\n t/helper/test-run-command.c | 4 +---\n 1 file changed, 1 insertion(+), 3 deletions(-)\n\ndiff --git a/t/helper/test-run-command.c b/t/helper/test-run-command.c\nindex a41a54d9cb..e6bd792274 100644\n--- a/t/helper/test-run-command.c\n+++ b/t/helper/test-run-command.c\n@@ -175,9 +175,7 @@ static int testsuite(int argc, const char **argv)\n \twhile ((d = readdir(dir))) {\n \t\tconst char *p = d->d_name;\n \n-\t\tif (*p != 't' || !isdigit(p[1]) || !isdigit(p[2]) ||\n-\t\t !isdigit(p[3]) || !isdigit(p[4]) || p[5] != '-' ||\n-\t\t !ends_with(p, \".sh\"))\n+\t\tif (!strcmp(p, \".\") || !strcmp(p, \"..\"))\n \t\t\tcontinue;\n \n \t\t/* No pattern: match all */\n-- \n2.45.0.rc0.197.gbae5840b3b-goog\n\n"},{"id":"493812","messageId":"abc9a7afe8c1ab92bfbd2b9eea672bdd9ff3c107.1714506612.git.steadmon@google.com","threadId":"60754","inReplyTo":"cover.1714506612.git.steadmon@google.com","subject":"[PATCH v5 4/7] test-tool run-command testsuite: support unit tests","fromName":"Josh Steadmon","fromEmail":"steadmon@google.com","sentAt":"2024-04-30T19:55:54Z","receivedAt":"2024-04-30T19:56:08Z","isPatch":true,"sender":{"key":"steadmon@google.com","avatar":"https://avatars.githubusercontent.com/u/2654920?v=4"},"body":"Teach the testsuite runner in `test-tool run-command testsuite` how to\nrun unit tests: if TEST_SHELL_PATH is not set, run the programs directly\nfrom CWD, rather than defaulting to \"sh\" as an interpreter.\n\nWith this change, you can now use test-tool to run the unit tests:\n$ make\n$ cd t/unit-tests/bin\n$ ../../helper/test-tool run-command testsuite\n\nThis should be helpful on Windows to allow running tests without\nrequiring Perl (for `prove`), as discussed in [1] and [2].\n\nThis again breaks backwards compatibility, as it is now required to set\nTEST_SHELL_PATH properly for executing shell scripts, but again, as\nnoted in [2], there are no longer any such invocations in our codebase.\n\n[1] https://lore.kernel.org/git/nycvar.QRO.7.76.6.2109091323150.59@tvgsbejvaqbjf.bet/\n[2] https://lore.kernel.org/git/850ea42c-f103-68d5-896b-9120e2628686@gmx.de/\n\nSigned-off-by: Josh Steadmon \n---\n t/helper/test-run-command.c | 17 ++++++++++++++---\n 1 file changed, 14 insertions(+), 3 deletions(-)\n\ndiff --git a/t/helper/test-run-command.c b/t/helper/test-run-command.c\nindex e6bd792274..61eb1175fe 100644\n--- a/t/helper/test-run-command.c\n+++ b/t/helper/test-run-command.c\n@@ -158,6 +158,8 @@ static int testsuite(int argc, const char **argv)\n \t\t.task_finished = test_finished,\n \t\t.data = &suite,\n \t};\n+\tstruct strbuf progpath = STRBUF_INIT;\n+\tsize_t path_prefix_len;\n \n \targc = parse_options(argc, argv, NULL, options,\n \t\t\ttestsuite_usage, PARSE_OPT_STOP_AT_NON_OPTION);\n@@ -165,9 +167,13 @@ static int testsuite(int argc, const char **argv)\n \tif (max_jobs <= 0)\n \t\tmax_jobs = online_cpus();\n \n+\t/*\n+\t * If we run without a shell, execute the programs directly from CWD.\n+\t */\n \tsuite.shell_path = getenv(\"TEST_SHELL_PATH\");\n \tif (!suite.shell_path)\n-\t\tsuite.shell_path = \"sh\";\n+\t\tstrbuf_addstr(&progpath, \"./\");\n+\tpath_prefix_len = progpath.len;\n \n \tdir = opendir(\".\");\n \tif (!dir)\n@@ -180,13 +186,17 @@ static int testsuite(int argc, const char **argv)\n \n \t\t/* No pattern: match all */\n \t\tif (!argc) {\n-\t\t\tstring_list_append(&suite.tests, p);\n+\t\t\tstrbuf_setlen(&progpath, path_prefix_len);\n+\t\t\tstrbuf_addstr(&progpath, p);\n+\t\t\tstring_list_append(&suite.tests, progpath.buf);\n \t\t\tcontinue;\n \t\t}\n \n \t\tfor (i = 0; i < argc; i++)\n \t\t\tif (!wildmatch(argv[i], p, 0)) {\n-\t\t\t\tstring_list_append(&suite.tests, p);\n+\t\t\t\tstrbuf_setlen(&progpath, path_prefix_len);\n+\t\t\t\tstrbuf_addstr(&progpath, p);\n+\t\t\t\tstring_list_append(&suite.tests, progpath.buf);\n \t\t\t\tbreak;\n \t\t\t}\n \t}\n@@ -213,6 +223,7 @@ static int testsuite(int argc, const char **argv)\n \n \tstring_list_clear(&suite.tests, 0);\n \tstring_list_clear(&suite.failed, 0);\n+\tstrbuf_release(&progpath);\n \n \treturn ret;\n }\n-- \n2.45.0.rc0.197.gbae5840b3b-goog\n\n"},{"id":"493813","messageId":"a8bbff2c6bcea6a8874d2a145cffe370d6cbddfb.1714506612.git.steadmon@google.com","threadId":"60754","inReplyTo":"cover.1714506612.git.steadmon@google.com","subject":"[PATCH v5 5/7] unit tests: add rule for running with test-tool","fromName":"Josh Steadmon","fromEmail":"steadmon@google.com","sentAt":"2024-04-30T19:55:55Z","receivedAt":"2024-04-30T19:56:10Z","isPatch":true,"sender":{"key":"steadmon@google.com","avatar":"https://avatars.githubusercontent.com/u/2654920?v=4"},"body":"In the previous commit, we added support in test-tool for running\ncollections of unit tests. Now, add rules in t/Makefile for running in\nthis way.\n\nThis new rule can be executed from the top-level Makefile via\n`make DEFAULT_UNIT_TEST_TARGET=unit-tests-test-tool unit-tests`, or by\nsetting DEFAULT_UNIT_TEST_TARGET in config.mak.\n\nSigned-off-by: Josh Steadmon \n---\n Makefile | 2 +-\n t/Makefile | 10 +++++++++-\n 2 files changed, 10 insertions(+), 2 deletions(-)\n\ndiff --git a/Makefile b/Makefile\nindex ba55d817ee..b0d1f04b4d 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -3870,5 +3870,5 @@ $(UNIT_TEST_PROGS): $(UNIT_TEST_BIN)/%$X: $(UNIT_TEST_DIR)/%.o $(UNIT_TEST_DIR)/\n \n .PHONY: build-unit-tests unit-tests\n build-unit-tests: $(UNIT_TEST_PROGS)\n-unit-tests: $(UNIT_TEST_PROGS)\n+unit-tests: $(UNIT_TEST_PROGS) t/helper/test-tool$X\n \t$(MAKE) -C t/ unit-tests\ndiff --git a/t/Makefile b/t/Makefile\nindex 4861edafe6..0ae04f1e42 100644\n--- a/t/Makefile\n+++ b/t/Makefile\n@@ -49,6 +49,7 @@ CHAINLINT = '$(PERL_PATH_SQ)' chainlint.pl\n UNIT_TEST_SOURCES = $(wildcard unit-tests/t-*.c)\n UNIT_TEST_PROGRAMS = $(patsubst unit-tests/%.c,unit-tests/bin/%$(X),$(UNIT_TEST_SOURCES))\n UNIT_TESTS = $(sort $(UNIT_TEST_PROGRAMS))\n+UNIT_TESTS_NO_DIR = $(notdir $(UNIT_TESTS))\n \n # `test-chainlint` (which is a dependency of `test-lint`, `test` and `prove`)\n # checks all tests in all scripts via a single invocation, so tell individual\n@@ -76,7 +77,7 @@ $(T):\n $(UNIT_TESTS):\n \t@echo \"*** $@ ***\"; $@\n \n-.PHONY: unit-tests unit-tests-raw unit-tests-prove\n+.PHONY: unit-tests unit-tests-raw unit-tests-prove unit-tests-test-tool\n unit-tests: $(DEFAULT_UNIT_TEST_TARGET)\n \n unit-tests-raw: $(UNIT_TESTS)\n@@ -84,6 +85,13 @@ unit-tests-raw: $(UNIT_TESTS)\n unit-tests-prove:\n \t@echo \"*** prove - unit tests ***\"; $(PROVE) $(GIT_PROVE_OPTS) $(UNIT_TESTS)\n \n+unit-tests-test-tool:\n+\t@echo \"*** test-tool - unit tests **\"\n+\t( \\\n+\t\tcd unit-tests/bin && \\\n+\t\t../../helper/test-tool$X run-command testsuite $(UNIT_TESTS_NO_DIR)\\\n+\t)\n+\n pre-clean:\n \t$(RM) -r '$(TEST_RESULTS_DIRECTORY_SQ)'\n \n-- \n2.45.0.rc0.197.gbae5840b3b-goog\n\n"},{"id":"493814","messageId":"c6606446c47a7f49007e058e9ba84025919d86c4.1714506612.git.steadmon@google.com","threadId":"60754","inReplyTo":"cover.1714506612.git.steadmon@google.com","subject":"[PATCH v5 6/7] t/Makefile: run unit tests alongside shell tests","fromName":"Josh Steadmon","fromEmail":"steadmon@google.com","sentAt":"2024-04-30T19:55:56Z","receivedAt":"2024-04-30T19:56:12Z","isPatch":true,"sender":{"key":"steadmon@google.com","avatar":"https://avatars.githubusercontent.com/u/2654920?v=4"},"body":"From: Jeff King \n\nAdd a wrapper script to allow `prove` to run both shell tests and unit\ntests from a single invocation. This avoids issues around running prove\ntwice in CI, as discussed in [1].\n\nAdditionally, this moves the unit tests into the main dev workflow, so\nthat errors can be spotted more quickly. Accordingly, we remove the\nseparate unit tests step for Linux CI. (We leave the Windows CI\nunit-test step as-is, because the sharding scheme there involves\nselecting specific test files rather than running `make test`.)\n\n[1] https://lore.kernel.org/git/pull.1613.git.1699894837844.gitgitgadget@gmail.com/\n\nSigned-off-by: Jeff King \nSigned-off-by: Josh Steadmon \n---\n ci/run-build-and-tests.sh | 2 --\n t/Makefile | 2 +-\n t/run-test.sh | 18 ++++++++++++++++++\n 3 files changed, 19 insertions(+), 3 deletions(-)\n create mode 100755 t/run-test.sh\n\ndiff --git a/ci/run-build-and-tests.sh b/ci/run-build-and-tests.sh\nindex 7a1466b868..2528f25e31 100755\n--- a/ci/run-build-and-tests.sh\n+++ b/ci/run-build-and-tests.sh\n@@ -50,8 +50,6 @@ if test -n \"$run_tests\"\n then\n \tgroup \"Run tests\" make test ||\n \thandle_failed_tests\n-\tgroup \"Run unit tests\" \\\n-\t\tmake DEFAULT_UNIT_TEST_TARGET=unit-tests-prove unit-tests\n fi\n check_unignored_build_artifacts\n \ndiff --git a/t/Makefile b/t/Makefile\nindex 0ae04f1e42..b2eb9f770b 100644\n--- a/t/Makefile\n+++ b/t/Makefile\n@@ -68,7 +68,7 @@ failed:\n \ttest -z \"$$failed\" || $(MAKE) $$failed\n \n prove: pre-clean check-chainlint $(TEST_LINT)\n-\t@echo \"*** prove ***\"; $(CHAINLINTSUPPRESS) $(PROVE) --exec '$(TEST_SHELL_PATH_SQ)' $(GIT_PROVE_OPTS) $(T) :: $(GIT_TEST_OPTS)\n+\t@echo \"*** prove (shell & unit tests) ***\"; $(CHAINLINTSUPPRESS) TEST_SHELL_PATH='$(TEST_SHELL_PATH_SQ)' $(PROVE) --exec ./run-test.sh $(GIT_PROVE_OPTS) $(T) $(UNIT_TESTS) :: $(GIT_TEST_OPTS)\n \t$(MAKE) clean-except-prove-cache\n \n $(T):\ndiff --git a/t/run-test.sh b/t/run-test.sh\nnew file mode 100755\nindex 0000000000..0eabf42d69\n--- /dev/null\n+++ b/t/run-test.sh\n@@ -0,0 +1,18 @@\n+#!/bin/sh\n+\n+# A simple wrapper to run shell tests via TEST_SHELL_PATH,\n+# or exec unit tests directly.\n+\n+case \"$1\" in\n+*.sh)\n+\tif test -z \"${TEST_SHELL_PATH+set}\"\n+\tthen\n+\t\techo >&2 \"ERROR: TEST_SHELL_PATH is empty or not set\"\n+\t\texit 1\n+\tfi\n+\texec ${TEST_SHELL_PATH} \"$@\"\n+\t;;\n+*)\n+\texec \"$@\"\n+\t;;\n+esac\n-- \n2.45.0.rc0.197.gbae5840b3b-goog\n\n"},{"id":"493815","messageId":"4a92131ab94b06958db2d2172344d427f0494b83.1714506612.git.steadmon@google.com","threadId":"60754","inReplyTo":"cover.1714506612.git.steadmon@google.com","subject":"[PATCH v5 7/7] ci: use test-tool as unit test runner on Windows","fromName":"Josh Steadmon","fromEmail":"steadmon@google.com","sentAt":"2024-04-30T19:55:57Z","receivedAt":"2024-04-30T19:56:13Z","isPatch":true,"sender":{"key":"steadmon@google.com","avatar":"https://avatars.githubusercontent.com/u/2654920?v=4"},"body":"Although the previous commit changed t/Makefile to run unit tests\nalongside shell tests, the Windows CI still needs a separate unit-tests\nstep due to how the test sharding works.\n\nWe want to avoid using `prove` as a test running on Windows due to\nperformance issues [1], so use the new test-tool runner instead.\n\n[1] https://lore.kernel.org/git/850ea42c-f103-68d5-896b-9120e2628686@gmx.de/\n\nSigned-off-by: Josh Steadmon \n---\n ci/run-test-slice.sh | 2 +-\n 1 file changed, 1 insertion(+), 1 deletion(-)\n\ndiff --git a/ci/run-test-slice.sh b/ci/run-test-slice.sh\nindex ae8094382f..e167e646f7 100755\n--- a/ci/run-test-slice.sh\n+++ b/ci/run-test-slice.sh\n@@ -17,7 +17,7 @@ handle_failed_tests\n \n # We only have one unit test at the moment, so run it in the first slice\n if [ \"$1\" == \"0\" ] ; then\n-\tgroup \"Run unit tests\" make --quiet -C t unit-tests-prove\n+\tgroup \"Run unit tests\" make --quiet -C t unit-tests-test-tool\n fi\n \n check_unignored_build_artifacts\n-- \n2.45.0.rc0.197.gbae5840b3b-goog\n\n"},{"id":"493819","messageId":"xmqqle4uy328.fsf@gitster.g","threadId":"60754","inReplyTo":"c6606446c47a7f49007e058e9ba84025919d86c4.1714506612.git.steadmon@google.com","subject":"Re: [PATCH v5 6/7] t/Makefile: run unit tests alongside shell tests","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2024-04-30T21:05:03Z","receivedAt":"2024-04-30T21:05:11Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Josh Steadmon writes:\n\n> +# A simple wrapper to run shell tests via TEST_SHELL_PATH,\n> +# or exec unit tests directly.\n> +\n> +case \"$1\" in\n> +*.sh)\n> +\tif test -z \"${TEST_SHELL_PATH+set}\"\n> +\tthen\n> +\t\techo >&2 \"ERROR: TEST_SHELL_PATH is empty or not set\"\n> +\t\texit 1\n> +\tfi\n> +\texec ${TEST_SHELL_PATH} \"$@\"\n\nYou want to be prepared for a path like \"//C/My Programs/bash\".\n\n\texec \"$TEST_SHELL_PATH\" \"$@\"\n\n> +\t;;\n> +*)\n> +\texec \"$@\"\n> +\t;;\n> +esac\n"},{"id":"493820","messageId":"xmqqbk5qy2l3.fsf@gitster.g","threadId":"60754","inReplyTo":"cover.1714506612.git.steadmon@google.com","subject":"Re: [PATCH v5 0/7] test-tool: add unit test suite runner","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2024-04-30T21:15:20Z","receivedAt":"2024-04-30T21:15:27Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Josh Steadmon writes:\n\n> For various reasons (see discussion at [1]) we would like an alternative\n> to `prove` for running test suites (including the unit tests) on\n> Windows.\n>\n> [1] https://lore.kernel.org/git/pull.1613.git.1699894837844.gitgitgadget@gmail.com/\n>\n> This series extends the existing `test-tool run-command testsuite` to\n> support running unit tests. In addition, it includes some small\n> cleanups:\n> * move t-basic out of the unit-tests directory\n> * don't hardcode the shell for running tests in `test-tool ... testsuite`\n> * don't hardcode a test name filter in `test-tool ... testsuite`\n> * add a test wrapper script to allow unit tests and the shell test suite\n> to run in a single `prove` process\n\nI am OK to see it outside the scope of this series, but we would\nneed unit tests supported by the GIT_SKIP_TESTS mechanism (or an\nalternative mechanism written), given that I hear \"migrate tests to\nunit-test framework\" every once in a while, which means we would\naccumlate more and more tests that ignore GIT_SKIP_TESTS mechansim.\n\nI did spot one potential problem (rather, \"we'd want to fix it as we\nare changing it with this reroll anyway\"), but other than that I did\nnot see anything wrong in the other patches.\n\nThanks. Will queue.\n"},{"id":"494036","messageId":"20240503180239.GA3634479@coredump.intra.peff.net","threadId":"60754","inReplyTo":"xmqqo79yxxk7.fsf@gitster.g","subject":"Re: [PATCH v4 6/7] t/Makefile: run unit tests alongside shell tests","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2024-05-03T18:02:39Z","receivedAt":"2024-05-03T18:02:40Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Wed, Apr 24, 2024 at 02:25:44PM -0700, Junio C Hamano wrote:\n\n> Josh Steadmon writes:\n> \n> > +case \"$1\" in\n> > +*.sh)\n> > +\tif test -z \"${TEST_SHELL_PATH+set}\" ; then\n> > +\t\techo \"ERROR: TEST_SHELL_PATH is not set\" >&2\n> \n> Style.\n> \n> As an empty string is not a reasonable value for this variable (and\n> you do not quote ${TEST_SHELL_PATH} when you use it in \"exec\" below),\n> \n> \tif test -z \"${TEST_SHELL_PATH:+set}\"\n> \tthen\n> \t\techo >&2 \"ERROR: TEST_SHELL_PATH is not set or empty\"\n> \n> may be what we want here.\n\nIf we are using \":+\" to handle the empty string, I think just:\n\n if test -z \"$TEST_SHELL_PATH\"\n\nis sufficient, no?\n\n(not that the other is incorrect, but whenever I see something like\n\":+set\" I wonder if something more clever is going on, and of course I\nget nightmare flashbacks to looking at generated autoconf code).\n\n-Peff\n"},{"id":"494048","messageId":"xmqq34qy7lie.fsf@gitster.g","threadId":"60754","inReplyTo":"20240503180239.GA3634479@coredump.intra.peff.net","subject":"Re: [PATCH v4 6/7] t/Makefile: run unit tests alongside shell tests","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2024-05-03T19:17:45Z","receivedAt":"2024-05-03T19:17:53Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Jeff King writes:\n\n>> \tif test -z \"${TEST_SHELL_PATH:+set}\"\n>> \tthen\n>> \t\techo >&2 \"ERROR: TEST_SHELL_PATH is not set or empty\"\n>> \n>> may be what we want here.\n>\n> If we are using \":+\" to handle the empty string, I think just:\n>\n> if test -z \"$TEST_SHELL_PATH\"\n>\n> is sufficient, no?\n\nYes. And the other part of this hunk still needs fixing, namely,\n\n> +\t\texit 1\n> +\tfi\n> +\texec ${TEST_SHELL_PATH} \"$@\"\n> +\t;;\n\nthe above reference needs to be quoted protect $IFS in the path.\n"},{"id":"494186","messageId":"cover.1715024899.git.steadmon@google.com","threadId":"60754","inReplyTo":"cover.1705443632.git.steadmon@google.com","subject":"[PATCH v6 0/7] test-tool: add unit test suite runner","fromName":"Josh Steadmon","fromEmail":"steadmon@google.com","sentAt":"2024-05-06T19:57:30Z","receivedAt":"2024-05-06T19:57:39Z","isPatch":true,"sender":{"key":"steadmon@google.com","avatar":"https://avatars.githubusercontent.com/u/2654920?v=4"},"body":"For various reasons (see discussion at [1]) we would like an alternative\nto `prove` for running test suites (including the unit tests) on\nWindows.\n\n[1] https://lore.kernel.org/git/pull.1613.git.1699894837844.gitgitgadget@gmail.com/\n\nThis series extends the existing `test-tool run-command testsuite` to\nsupport running unit tests. In addition, it includes some small\ncleanups:\n* move t-basic out of the unit-tests directory\n* don't hardcode the shell for running tests in `test-tool ... testsuite`\n* don't hardcode a test name filter in `test-tool ... testsuite`\n* add a test wrapper script to allow unit tests and the shell test suite\n to run in a single `prove` process\n\nChanges in V4:\n* Set TEST_SHELL_PATH when running the `prove` test target, and error\n out of run-test.sh if TEST_SHELL_PATH is not set.\n\nChanges in V3:\n* Added new patch (#7) to use the new test-tool runner for unit tests in\n Windows CI.\n* Restored the explicit sort call in t/Makefile, for backwards\n compatibility with older GNU Make versions.\n* Rebased the series on the latest jk/unit-tests-buildfix branch.\n* Fixed header include order in patch #1.\n* Removed a paragraph in patch #1's commit message that is obsolete now\n that we're building the list of test files from the sources rather\n than by globbing.\n* Added a note in patch #2 that setting a NULL suite.shell_path will be\n used in a later commit.\n* Clarified up some sloppy wording in commit messages and comments in\n t/helper/test-run-command.c.\n\nChanges in V2:\n* Rebased the series on the latest jk/unit-tests-buildfix branch.\n* Patch 1: move t-basic to a test-tool subcommand rather than a new\n executable under t/t0080/\n* New patch 2: get the shell path from TEST_SHELL_PATH in\n `test-tool run-command testsuite`\n* New patch 3: remove the hardcoded filename filter in\n `test-tool run-command testsuite`\n* Patch 4 (previously 2): simplified now that we no longer need to add\n any command-line flags to support unit tests\n* Patch 5 (previously 3): avoid trying to run cmake *.pdb files by using\n the unit test list built in the makefile in jk/unit-tests-buildfix.\n\n\nJeff King (1):\n t/Makefile: run unit tests alongside shell tests\n\nJosh Steadmon (6):\n t0080: turn t-basic unit test into a helper\n test-tool run-command testsuite: get shell from env\n test-tool run-command testsuite: remove hardcoded filter\n test-tool run-command testsuite: support unit tests\n unit tests: add rule for running with test-tool\n ci: use test-tool as unit test runner on Windows\n\n Makefile | 6 ++--\n ci/run-build-and-tests.sh | 2 --\n ci/run-test-slice.sh | 2 +-\n t/Makefile | 14 ++++++++--\n .../t-basic.c => helper/test-example-tap.c} | 5 ++--\n t/helper/test-run-command.c | 28 +++++++++++++++----\n t/helper/test-tool.c | 1 +\n t/helper/test-tool.h | 1 +\n t/run-test.sh | 18 ++++++++++++\n t/t0080-unit-test-output.sh | 24 ++++++++--------\n 10 files changed, 72 insertions(+), 29 deletions(-)\n rename t/{unit-tests/t-basic.c => helper/test-example-tap.c} (95%)\n create mode 100755 t/run-test.sh\n\nRange-diff against v5:\n1: 6777451100 = 1: 6777451100 t0080: turn t-basic unit test into a helper\n2: 24f47f8fc7 = 2: 24f47f8fc7 test-tool run-command testsuite: get shell from env\n3: 4a16a3ec24 = 3: 4a16a3ec24 test-tool run-command testsuite: remove hardcoded filter\n4: abc9a7afe8 = 4: abc9a7afe8 test-tool run-command testsuite: support unit tests\n5: a8bbff2c6b = 5: a8bbff2c6b unit tests: add rule for running with test-tool\n6: c6606446c4 ! 6: dae670fcb1 t/Makefile: run unit tests alongside shell tests\n @@ t/run-test.sh (new)\n +\n +case \"$1\" in\n +*.sh)\n -+ if test -z \"${TEST_SHELL_PATH+set}\"\n ++ if test -z \"${TEST_SHELL_PATH}\"\n + then\n + echo >&2 \"ERROR: TEST_SHELL_PATH is empty or not set\"\n + exit 1\n + fi\n -+ exec ${TEST_SHELL_PATH} \"$@\"\n ++ exec \"${TEST_SHELL_PATH}\" \"$@\"\n + ;;\n +*)\n + exec \"$@\"\n7: 4a92131ab9 = 7: 2f5853a7fa ci: use test-tool as unit test runner on Windows\n\nbase-commit: 4904a4d08cc085716df12ce713ae7ee3d5ecb75a\n-- \n2.45.0.rc1.225.g2a3ae87e7f-goog\n\n"},{"id":"494187","messageId":"677745110078501e611faf3d1facd1518e0453a2.1715024899.git.steadmon@google.com","threadId":"60754","inReplyTo":"cover.1715024899.git.steadmon@google.com","subject":"[PATCH v6 1/7] t0080: turn t-basic unit test into a helper","fromName":"Josh Steadmon","fromEmail":"steadmon@google.com","sentAt":"2024-05-06T19:57:31Z","receivedAt":"2024-05-06T19:57:41Z","isPatch":true,"sender":{"key":"steadmon@google.com","avatar":"https://avatars.githubusercontent.com/u/2654920?v=4"},"body":"While t/unit-tests/t-basic.c uses the unit-test framework added in\ne137fe3b29 (unit tests: add TAP unit test framework, 2023-11-09), it is\nnot a true unit test in that it intentionally fails in order to exercise\nvarious codepaths in the unit-test framework. Thus, we intentionally\nexclude it when running unit tests through the various t/Makefile\ntargets. Instead, it is executed by t0080-unit-test-output.sh, which\nverifies its output follows the TAP format expected for the various\npass, skip, or fail cases.\n\nAs such, it makes more sense for t-basic to be a helper item for\nt0080-unit-test-output.sh, so let's move it to\nt/helper/test-example-tap.c and adjust Makefiles as necessary.\n\nSigned-off-by: Josh Steadmon \n---\n Makefile | 4 ++--\n t/Makefile | 2 +-\n .../t-basic.c => helper/test-example-tap.c} | 5 ++--\n t/helper/test-tool.c | 1 +\n t/helper/test-tool.h | 1 +\n t/t0080-unit-test-output.sh | 24 +++++++++----------\n 6 files changed, 20 insertions(+), 17 deletions(-)\n rename t/{unit-tests/t-basic.c => helper/test-example-tap.c} (95%)\n\ndiff --git a/Makefile b/Makefile\nindex 23723367b8..ba55d817ee 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -802,6 +802,7 @@ TEST_BUILTINS_OBJS += test-dump-split-index.o\n TEST_BUILTINS_OBJS += test-dump-untracked-cache.o\n TEST_BUILTINS_OBJS += test-env-helper.o\n TEST_BUILTINS_OBJS += test-example-decorate.o\n+TEST_BUILTINS_OBJS += test-example-tap.o\n TEST_BUILTINS_OBJS += test-find-pack.o\n TEST_BUILTINS_OBJS += test-fsmonitor-client.o\n TEST_BUILTINS_OBJS += test-genrandom.o\n@@ -1338,7 +1339,6 @@ THIRD_PARTY_SOURCES += compat/regex/%\n THIRD_PARTY_SOURCES += sha1collisiondetection/%\n THIRD_PARTY_SOURCES += sha1dc/%\n \n-UNIT_TEST_PROGRAMS += t-basic\n UNIT_TEST_PROGRAMS += t-mem-pool\n UNIT_TEST_PROGRAMS += t-strbuf\n UNIT_TEST_PROGRAMS += t-ctype\n@@ -3218,7 +3218,7 @@ perf: all\n \n .PRECIOUS: $(TEST_OBJS)\n \n-t/helper/test-tool$X: $(patsubst %,t/helper/%,$(TEST_BUILTINS_OBJS))\n+t/helper/test-tool$X: $(patsubst %,t/helper/%,$(TEST_BUILTINS_OBJS)) $(UNIT_TEST_DIR)/test-lib.o\n \n t/helper/test-%$X: t/helper/test-%.o GIT-LDFLAGS $(GITLIBS) $(REFTABLE_TEST_LIB)\n \t$(QUIET_LINK)$(CC) $(ALL_CFLAGS) -o $@ $(ALL_LDFLAGS) $(filter %.o,$^) $(filter %.a,$^) $(LIBS)\ndiff --git a/t/Makefile b/t/Makefile\nindex 2d95046f26..4861edafe6 100644\n--- a/t/Makefile\n+++ b/t/Makefile\n@@ -48,7 +48,7 @@ CHAINLINTTESTS = $(sort $(patsubst chainlint/%.test,%,$(wildcard chainlint/*.tes\n CHAINLINT = '$(PERL_PATH_SQ)' chainlint.pl\n UNIT_TEST_SOURCES = $(wildcard unit-tests/t-*.c)\n UNIT_TEST_PROGRAMS = $(patsubst unit-tests/%.c,unit-tests/bin/%$(X),$(UNIT_TEST_SOURCES))\n-UNIT_TESTS = $(sort $(filter-out unit-tests/bin/t-basic%,$(UNIT_TEST_PROGRAMS)))\n+UNIT_TESTS = $(sort $(UNIT_TEST_PROGRAMS))\n \n # `test-chainlint` (which is a dependency of `test-lint`, `test` and `prove`)\n # checks all tests in all scripts via a single invocation, so tell individual\ndiff --git a/t/unit-tests/t-basic.c b/t/helper/test-example-tap.c\nsimilarity index 95%\nrename from t/unit-tests/t-basic.c\nrename to t/helper/test-example-tap.c\nindex fda1ae59a6..d072ad559f 100644\n--- a/t/unit-tests/t-basic.c\n+++ b/t/helper/test-example-tap.c\n@@ -1,4 +1,5 @@\n-#include \"test-lib.h\"\n+#include \"test-tool.h\"\n+#include \"t/unit-tests/test-lib.h\"\n \n /*\n * The purpose of this \"unit test\" is to verify a few invariants of the unit\n@@ -69,7 +70,7 @@ static void t_empty(void)\n \t; /* empty */\n }\n \n-int cmd_main(int argc, const char **argv)\n+int cmd__example_tap(int argc, const char **argv)\n {\n \ttest_res = TEST(check_res = check_int(1, ==, 1), \"passing test\");\n \tTEST(t_res(1), \"passing test and assertion return 1\");\ndiff --git a/t/helper/test-tool.c b/t/helper/test-tool.c\nindex 33b9501c21..bb5c04c9c0 100644\n--- a/t/helper/test-tool.c\n+++ b/t/helper/test-tool.c\n@@ -29,6 +29,7 @@ static struct test_cmd cmds[] = {\n \t{ \"dump-untracked-cache\", cmd__dump_untracked_cache },\n \t{ \"env-helper\", cmd__env_helper },\n \t{ \"example-decorate\", cmd__example_decorate },\n+\t{ \"example-tap\", cmd__example_tap },\n \t{ \"find-pack\", cmd__find_pack },\n \t{ \"fsmonitor-client\", cmd__fsmonitor_client },\n \t{ \"genrandom\", cmd__genrandom },\ndiff --git a/t/helper/test-tool.h b/t/helper/test-tool.h\nindex b72f07ded9..38001bd1c6 100644\n--- a/t/helper/test-tool.h\n+++ b/t/helper/test-tool.h\n@@ -23,6 +23,7 @@ int cmd__dump_untracked_cache(int argc, const char **argv);\n int cmd__dump_reftable(int argc, const char **argv);\n int cmd__env_helper(int argc, const char **argv);\n int cmd__example_decorate(int argc, const char **argv);\n+int cmd__example_tap(int argc, const char **argv);\n int cmd__find_pack(int argc, const char **argv);\n int cmd__fsmonitor_client(int argc, const char **argv);\n int cmd__genrandom(int argc, const char **argv);\ndiff --git a/t/t0080-unit-test-output.sh b/t/t0080-unit-test-output.sh\nindex 961b54b06c..83b1e3b7f5 100755\n--- a/t/t0080-unit-test-output.sh\n+++ b/t/t0080-unit-test-output.sh\n@@ -8,50 +8,50 @@ test_expect_success 'TAP output from unit tests' '\n \tcat >expect <<-EOF &&\n \tok 1 - passing test\n \tok 2 - passing test and assertion return 1\n-\t# check \"1 == 2\" failed at t/unit-tests/t-basic.c:76\n+\t# check \"1 == 2\" failed at t/helper/test-example-tap.c:77\n \t# left: 1\n \t# right: 2\n \tnot ok 3 - failing test\n \tok 4 - failing test and assertion return 0\n \tnot ok 5 - passing TEST_TODO() # TODO\n \tok 6 - passing TEST_TODO() returns 1\n-\t# todo check ${SQ}check(x)${SQ} succeeded at t/unit-tests/t-basic.c:25\n+\t# todo check ${SQ}check(x)${SQ} succeeded at t/helper/test-example-tap.c:26\n \tnot ok 7 - failing TEST_TODO()\n \tok 8 - failing TEST_TODO() returns 0\n-\t# check \"0\" failed at t/unit-tests/t-basic.c:30\n+\t# check \"0\" failed at t/helper/test-example-tap.c:31\n \t# skipping test - missing prerequisite\n-\t# skipping check ${SQ}1${SQ} at t/unit-tests/t-basic.c:32\n+\t# skipping check ${SQ}1${SQ} at t/helper/test-example-tap.c:33\n \tok 9 - test_skip() # SKIP\n \tok 10 - skipped test returns 1\n \t# skipping test - missing prerequisite\n \tok 11 - test_skip() inside TEST_TODO() # SKIP\n \tok 12 - test_skip() inside TEST_TODO() returns 1\n-\t# check \"0\" failed at t/unit-tests/t-basic.c:48\n+\t# check \"0\" failed at t/helper/test-example-tap.c:49\n \tnot ok 13 - TEST_TODO() after failing check\n \tok 14 - TEST_TODO() after failing check returns 0\n-\t# check \"0\" failed at t/unit-tests/t-basic.c:56\n+\t# check \"0\" failed at t/helper/test-example-tap.c:57\n \tnot ok 15 - failing check after TEST_TODO()\n \tok 16 - failing check after TEST_TODO() returns 0\n-\t# check \"!strcmp(\"\\thello\\\\\\\\\", \"there\\\"\\n\")\" failed at t/unit-tests/t-basic.c:61\n+\t# check \"!strcmp(\"\\thello\\\\\\\\\", \"there\\\"\\n\")\" failed at t/helper/test-example-tap.c:62\n \t# left: \"\\011hello\\\\\\\\\"\n \t# right: \"there\\\"\\012\"\n-\t# check \"!strcmp(\"NULL\", NULL)\" failed at t/unit-tests/t-basic.c:62\n+\t# check \"!strcmp(\"NULL\", NULL)\" failed at t/helper/test-example-tap.c:63\n \t# left: \"NULL\"\n \t# right: NULL\n-\t# check \"${SQ}a${SQ} == ${SQ}\\n${SQ}\" failed at t/unit-tests/t-basic.c:63\n+\t# check \"${SQ}a${SQ} == ${SQ}\\n${SQ}\" failed at t/helper/test-example-tap.c:64\n \t# left: ${SQ}a${SQ}\n \t# right: ${SQ}\\012${SQ}\n-\t# check \"${SQ}\\\\\\\\${SQ} == ${SQ}\\\\${SQ}${SQ}\" failed at t/unit-tests/t-basic.c:64\n+\t# check \"${SQ}\\\\\\\\${SQ} == ${SQ}\\\\${SQ}${SQ}\" failed at t/helper/test-example-tap.c:65\n \t# left: ${SQ}\\\\\\\\${SQ}\n \t# right: ${SQ}\\\\${SQ}${SQ}\n \tnot ok 17 - messages from failing string and char comparison\n-\t# BUG: test has no checks at t/unit-tests/t-basic.c:91\n+\t# BUG: test has no checks at t/helper/test-example-tap.c:92\n \tnot ok 18 - test with no checks\n \tok 19 - test with no checks returns 0\n \t1..19\n \tEOF\n \n-\t! \"$GIT_BUILD_DIR\"/t/unit-tests/bin/t-basic >actual &&\n+\t! test-tool example-tap >actual &&\n \ttest_cmp expect actual\n '\n \n-- \n2.45.0.rc1.225.g2a3ae87e7f-goog\n\n"},{"id":"494188","messageId":"24f47f8fc7e931c2e15fa985635e3036ec1123ba.1715024899.git.steadmon@google.com","threadId":"60754","inReplyTo":"cover.1715024899.git.steadmon@google.com","subject":"[PATCH v6 2/7] test-tool run-command testsuite: get shell from env","fromName":"Josh Steadmon","fromEmail":"steadmon@google.com","sentAt":"2024-05-06T19:57:32Z","receivedAt":"2024-05-06T19:57:43Z","isPatch":true,"sender":{"key":"steadmon@google.com","avatar":"https://avatars.githubusercontent.com/u/2654920?v=4"},"body":"When running tests through `test-tool run-command testsuite`, we\ncurrently hardcode `sh` as the command interpreter. As discussed in [1],\nthis is incorrect, and we should be using the shell set in\nTEST_SHELL_PATH instead.\n\nAdd a shell_path field in struct testsuite so that we can pass this to\nthe task runner callback. If this is non-null, we'll use it as the\nargv[0] of the subprocess. Otherwise, we'll just execute the test\nprogram directly. We will use this feature in a later commit to enable\nrunning binary executable unit tests.\n\nHowever, for now when setting up the struct testsuite in testsuite(),\nuse the value of TEST_SHELL_PATH if it's set, otherwise keep the\noriginal behavior by defaulting to `sh`.\n\n[1] https://lore.kernel.org/git/20240123005913.GB835964@coredump.intra.peff.net/\n\nSigned-off-by: Josh Steadmon \n---\n t/helper/test-run-command.c | 9 ++++++++-\n 1 file changed, 8 insertions(+), 1 deletion(-)\n\ndiff --git a/t/helper/test-run-command.c b/t/helper/test-run-command.c\nindex c0ed8722c8..a41a54d9cb 100644\n--- a/t/helper/test-run-command.c\n+++ b/t/helper/test-run-command.c\n@@ -65,6 +65,7 @@ struct testsuite {\n \tstruct string_list tests, failed;\n \tint next;\n \tint quiet, immediate, verbose, verbose_log, trace, write_junit_xml;\n+\tconst char *shell_path;\n };\n #define TESTSUITE_INIT { \\\n \t.tests = STRING_LIST_INIT_DUP, \\\n@@ -80,7 +81,9 @@ static int next_test(struct child_process *cp, struct strbuf *err, void *cb,\n \t\treturn 0;\n \n \ttest = suite->tests.items[suite->next++].string;\n-\tstrvec_pushl(&cp->args, \"sh\", test, NULL);\n+\tif (suite->shell_path)\n+\t\tstrvec_push(&cp->args, suite->shell_path);\n+\tstrvec_push(&cp->args, test);\n \tif (suite->quiet)\n \t\tstrvec_push(&cp->args, \"--quiet\");\n \tif (suite->immediate)\n@@ -162,6 +165,10 @@ static int testsuite(int argc, const char **argv)\n \tif (max_jobs <= 0)\n \t\tmax_jobs = online_cpus();\n \n+\tsuite.shell_path = getenv(\"TEST_SHELL_PATH\");\n+\tif (!suite.shell_path)\n+\t\tsuite.shell_path = \"sh\";\n+\n \tdir = opendir(\".\");\n \tif (!dir)\n \t\tdie(\"Could not open the current directory\");\n-- \n2.45.0.rc1.225.g2a3ae87e7f-goog\n\n"},{"id":"494189","messageId":"4a16a3ec241d83b3de424ce835db24eee6e46f0f.1715024899.git.steadmon@google.com","threadId":"60754","inReplyTo":"cover.1715024899.git.steadmon@google.com","subject":"[PATCH v6 3/7] test-tool run-command testsuite: remove hardcoded filter","fromName":"Josh Steadmon","fromEmail":"steadmon@google.com","sentAt":"2024-05-06T19:57:33Z","receivedAt":"2024-05-06T19:57:45Z","isPatch":true,"sender":{"key":"steadmon@google.com","avatar":"https://avatars.githubusercontent.com/u/2654920?v=4"},"body":"`test-tool run-command testsuite` currently assumes that it will only be\nrunning the shell test suite, and therefore filters out anything that\ndoes not match a hardcoded pattern of \"t[0-9][0-9][0-9][0-9]-*.sh\".\n\nLater in this series, we'll adapt `test-tool run-command testsuite` to\nalso support unit tests, which do not follow the same naming conventions\nas the shell tests, so this hardcoded pattern is inconvenient.\n\nSince `testsuite` also allows specifying patterns on the command-line,\nlet's just remove this pattern. As noted in [1], there are no longer any\nuses of `testsuite` in our codebase, it should be OK to break backwards\ncompatibility in this case. We also add a new filter to avoid trying to\nexecute \".\" and \"..\", so that users who wish to execute every test in a\ndirectory can do so without specifying a pattern.\n\n[1] https://lore.kernel.org/git/850ea42c-f103-68d5-896b-9120e2628686@gmx.de/\n\nSigned-off-by: Josh Steadmon \n---\n t/helper/test-run-command.c | 4 +---\n 1 file changed, 1 insertion(+), 3 deletions(-)\n\ndiff --git a/t/helper/test-run-command.c b/t/helper/test-run-command.c\nindex a41a54d9cb..e6bd792274 100644\n--- a/t/helper/test-run-command.c\n+++ b/t/helper/test-run-command.c\n@@ -175,9 +175,7 @@ static int testsuite(int argc, const char **argv)\n \twhile ((d = readdir(dir))) {\n \t\tconst char *p = d->d_name;\n \n-\t\tif (*p != 't' || !isdigit(p[1]) || !isdigit(p[2]) ||\n-\t\t !isdigit(p[3]) || !isdigit(p[4]) || p[5] != '-' ||\n-\t\t !ends_with(p, \".sh\"))\n+\t\tif (!strcmp(p, \".\") || !strcmp(p, \"..\"))\n \t\t\tcontinue;\n \n \t\t/* No pattern: match all */\n-- \n2.45.0.rc1.225.g2a3ae87e7f-goog\n\n"},{"id":"494190","messageId":"abc9a7afe8c1ab92bfbd2b9eea672bdd9ff3c107.1715024899.git.steadmon@google.com","threadId":"60754","inReplyTo":"cover.1715024899.git.steadmon@google.com","subject":"[PATCH v6 4/7] test-tool run-command testsuite: support unit tests","fromName":"Josh Steadmon","fromEmail":"steadmon@google.com","sentAt":"2024-05-06T19:57:34Z","receivedAt":"2024-05-06T19:57:47Z","isPatch":true,"sender":{"key":"steadmon@google.com","avatar":"https://avatars.githubusercontent.com/u/2654920?v=4"},"body":"Teach the testsuite runner in `test-tool run-command testsuite` how to\nrun unit tests: if TEST_SHELL_PATH is not set, run the programs directly\nfrom CWD, rather than defaulting to \"sh\" as an interpreter.\n\nWith this change, you can now use test-tool to run the unit tests:\n$ make\n$ cd t/unit-tests/bin\n$ ../../helper/test-tool run-command testsuite\n\nThis should be helpful on Windows to allow running tests without\nrequiring Perl (for `prove`), as discussed in [1] and [2].\n\nThis again breaks backwards compatibility, as it is now required to set\nTEST_SHELL_PATH properly for executing shell scripts, but again, as\nnoted in [2], there are no longer any such invocations in our codebase.\n\n[1] https://lore.kernel.org/git/nycvar.QRO.7.76.6.2109091323150.59@tvgsbejvaqbjf.bet/\n[2] https://lore.kernel.org/git/850ea42c-f103-68d5-896b-9120e2628686@gmx.de/\n\nSigned-off-by: Josh Steadmon \n---\n t/helper/test-run-command.c | 17 ++++++++++++++---\n 1 file changed, 14 insertions(+), 3 deletions(-)\n\ndiff --git a/t/helper/test-run-command.c b/t/helper/test-run-command.c\nindex e6bd792274..61eb1175fe 100644\n--- a/t/helper/test-run-command.c\n+++ b/t/helper/test-run-command.c\n@@ -158,6 +158,8 @@ static int testsuite(int argc, const char **argv)\n \t\t.task_finished = test_finished,\n \t\t.data = &suite,\n \t};\n+\tstruct strbuf progpath = STRBUF_INIT;\n+\tsize_t path_prefix_len;\n \n \targc = parse_options(argc, argv, NULL, options,\n \t\t\ttestsuite_usage, PARSE_OPT_STOP_AT_NON_OPTION);\n@@ -165,9 +167,13 @@ static int testsuite(int argc, const char **argv)\n \tif (max_jobs <= 0)\n \t\tmax_jobs = online_cpus();\n \n+\t/*\n+\t * If we run without a shell, execute the programs directly from CWD.\n+\t */\n \tsuite.shell_path = getenv(\"TEST_SHELL_PATH\");\n \tif (!suite.shell_path)\n-\t\tsuite.shell_path = \"sh\";\n+\t\tstrbuf_addstr(&progpath, \"./\");\n+\tpath_prefix_len = progpath.len;\n \n \tdir = opendir(\".\");\n \tif (!dir)\n@@ -180,13 +186,17 @@ static int testsuite(int argc, const char **argv)\n \n \t\t/* No pattern: match all */\n \t\tif (!argc) {\n-\t\t\tstring_list_append(&suite.tests, p);\n+\t\t\tstrbuf_setlen(&progpath, path_prefix_len);\n+\t\t\tstrbuf_addstr(&progpath, p);\n+\t\t\tstring_list_append(&suite.tests, progpath.buf);\n \t\t\tcontinue;\n \t\t}\n \n \t\tfor (i = 0; i < argc; i++)\n \t\t\tif (!wildmatch(argv[i], p, 0)) {\n-\t\t\t\tstring_list_append(&suite.tests, p);\n+\t\t\t\tstrbuf_setlen(&progpath, path_prefix_len);\n+\t\t\t\tstrbuf_addstr(&progpath, p);\n+\t\t\t\tstring_list_append(&suite.tests, progpath.buf);\n \t\t\t\tbreak;\n \t\t\t}\n \t}\n@@ -213,6 +223,7 @@ static int testsuite(int argc, const char **argv)\n \n \tstring_list_clear(&suite.tests, 0);\n \tstring_list_clear(&suite.failed, 0);\n+\tstrbuf_release(&progpath);\n \n \treturn ret;\n }\n-- \n2.45.0.rc1.225.g2a3ae87e7f-goog\n\n"},{"id":"494191","messageId":"a8bbff2c6bcea6a8874d2a145cffe370d6cbddfb.1715024899.git.steadmon@google.com","threadId":"60754","inReplyTo":"cover.1715024899.git.steadmon@google.com","subject":"[PATCH v6 5/7] unit tests: add rule for running with test-tool","fromName":"Josh Steadmon","fromEmail":"steadmon@google.com","sentAt":"2024-05-06T19:57:35Z","receivedAt":"2024-05-06T19:57:48Z","isPatch":true,"sender":{"key":"steadmon@google.com","avatar":"https://avatars.githubusercontent.com/u/2654920?v=4"},"body":"In the previous commit, we added support in test-tool for running\ncollections of unit tests. Now, add rules in t/Makefile for running in\nthis way.\n\nThis new rule can be executed from the top-level Makefile via\n`make DEFAULT_UNIT_TEST_TARGET=unit-tests-test-tool unit-tests`, or by\nsetting DEFAULT_UNIT_TEST_TARGET in config.mak.\n\nSigned-off-by: Josh Steadmon \n---\n Makefile | 2 +-\n t/Makefile | 10 +++++++++-\n 2 files changed, 10 insertions(+), 2 deletions(-)\n\ndiff --git a/Makefile b/Makefile\nindex ba55d817ee..b0d1f04b4d 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -3870,5 +3870,5 @@ $(UNIT_TEST_PROGS): $(UNIT_TEST_BIN)/%$X: $(UNIT_TEST_DIR)/%.o $(UNIT_TEST_DIR)/\n \n .PHONY: build-unit-tests unit-tests\n build-unit-tests: $(UNIT_TEST_PROGS)\n-unit-tests: $(UNIT_TEST_PROGS)\n+unit-tests: $(UNIT_TEST_PROGS) t/helper/test-tool$X\n \t$(MAKE) -C t/ unit-tests\ndiff --git a/t/Makefile b/t/Makefile\nindex 4861edafe6..0ae04f1e42 100644\n--- a/t/Makefile\n+++ b/t/Makefile\n@@ -49,6 +49,7 @@ CHAINLINT = '$(PERL_PATH_SQ)' chainlint.pl\n UNIT_TEST_SOURCES = $(wildcard unit-tests/t-*.c)\n UNIT_TEST_PROGRAMS = $(patsubst unit-tests/%.c,unit-tests/bin/%$(X),$(UNIT_TEST_SOURCES))\n UNIT_TESTS = $(sort $(UNIT_TEST_PROGRAMS))\n+UNIT_TESTS_NO_DIR = $(notdir $(UNIT_TESTS))\n \n # `test-chainlint` (which is a dependency of `test-lint`, `test` and `prove`)\n # checks all tests in all scripts via a single invocation, so tell individual\n@@ -76,7 +77,7 @@ $(T):\n $(UNIT_TESTS):\n \t@echo \"*** $@ ***\"; $@\n \n-.PHONY: unit-tests unit-tests-raw unit-tests-prove\n+.PHONY: unit-tests unit-tests-raw unit-tests-prove unit-tests-test-tool\n unit-tests: $(DEFAULT_UNIT_TEST_TARGET)\n \n unit-tests-raw: $(UNIT_TESTS)\n@@ -84,6 +85,13 @@ unit-tests-raw: $(UNIT_TESTS)\n unit-tests-prove:\n \t@echo \"*** prove - unit tests ***\"; $(PROVE) $(GIT_PROVE_OPTS) $(UNIT_TESTS)\n \n+unit-tests-test-tool:\n+\t@echo \"*** test-tool - unit tests **\"\n+\t( \\\n+\t\tcd unit-tests/bin && \\\n+\t\t../../helper/test-tool$X run-command testsuite $(UNIT_TESTS_NO_DIR)\\\n+\t)\n+\n pre-clean:\n \t$(RM) -r '$(TEST_RESULTS_DIRECTORY_SQ)'\n \n-- \n2.45.0.rc1.225.g2a3ae87e7f-goog\n\n"},{"id":"494192","messageId":"dae670fcb115e6ee9a490b1965b09aeaaa5e8a4e.1715024899.git.steadmon@google.com","threadId":"60754","inReplyTo":"cover.1715024899.git.steadmon@google.com","subject":"[PATCH v6 6/7] t/Makefile: run unit tests alongside shell tests","fromName":"Josh Steadmon","fromEmail":"steadmon@google.com","sentAt":"2024-05-06T19:57:36Z","receivedAt":"2024-05-06T19:57:50Z","isPatch":true,"sender":{"key":"steadmon@google.com","avatar":"https://avatars.githubusercontent.com/u/2654920?v=4"},"body":"From: Jeff King \n\nAdd a wrapper script to allow `prove` to run both shell tests and unit\ntests from a single invocation. This avoids issues around running prove\ntwice in CI, as discussed in [1].\n\nAdditionally, this moves the unit tests into the main dev workflow, so\nthat errors can be spotted more quickly. Accordingly, we remove the\nseparate unit tests step for Linux CI. (We leave the Windows CI\nunit-test step as-is, because the sharding scheme there involves\nselecting specific test files rather than running `make test`.)\n\n[1] https://lore.kernel.org/git/pull.1613.git.1699894837844.gitgitgadget@gmail.com/\n\nSigned-off-by: Jeff King \nSigned-off-by: Josh Steadmon \n---\n ci/run-build-and-tests.sh | 2 --\n t/Makefile | 2 +-\n t/run-test.sh | 18 ++++++++++++++++++\n 3 files changed, 19 insertions(+), 3 deletions(-)\n create mode 100755 t/run-test.sh\n\ndiff --git a/ci/run-build-and-tests.sh b/ci/run-build-and-tests.sh\nindex 7a1466b868..2528f25e31 100755\n--- a/ci/run-build-and-tests.sh\n+++ b/ci/run-build-and-tests.sh\n@@ -50,8 +50,6 @@ if test -n \"$run_tests\"\n then\n \tgroup \"Run tests\" make test ||\n \thandle_failed_tests\n-\tgroup \"Run unit tests\" \\\n-\t\tmake DEFAULT_UNIT_TEST_TARGET=unit-tests-prove unit-tests\n fi\n check_unignored_build_artifacts\n \ndiff --git a/t/Makefile b/t/Makefile\nindex 0ae04f1e42..b2eb9f770b 100644\n--- a/t/Makefile\n+++ b/t/Makefile\n@@ -68,7 +68,7 @@ failed:\n \ttest -z \"$$failed\" || $(MAKE) $$failed\n \n prove: pre-clean check-chainlint $(TEST_LINT)\n-\t@echo \"*** prove ***\"; $(CHAINLINTSUPPRESS) $(PROVE) --exec '$(TEST_SHELL_PATH_SQ)' $(GIT_PROVE_OPTS) $(T) :: $(GIT_TEST_OPTS)\n+\t@echo \"*** prove (shell & unit tests) ***\"; $(CHAINLINTSUPPRESS) TEST_SHELL_PATH='$(TEST_SHELL_PATH_SQ)' $(PROVE) --exec ./run-test.sh $(GIT_PROVE_OPTS) $(T) $(UNIT_TESTS) :: $(GIT_TEST_OPTS)\n \t$(MAKE) clean-except-prove-cache\n \n $(T):\ndiff --git a/t/run-test.sh b/t/run-test.sh\nnew file mode 100755\nindex 0000000000..13c353b91b\n--- /dev/null\n+++ b/t/run-test.sh\n@@ -0,0 +1,18 @@\n+#!/bin/sh\n+\n+# A simple wrapper to run shell tests via TEST_SHELL_PATH,\n+# or exec unit tests directly.\n+\n+case \"$1\" in\n+*.sh)\n+\tif test -z \"${TEST_SHELL_PATH}\"\n+\tthen\n+\t\techo >&2 \"ERROR: TEST_SHELL_PATH is empty or not set\"\n+\t\texit 1\n+\tfi\n+\texec \"${TEST_SHELL_PATH}\" \"$@\"\n+\t;;\n+*)\n+\texec \"$@\"\n+\t;;\n+esac\n-- \n2.45.0.rc1.225.g2a3ae87e7f-goog\n\n"},{"id":"494193","messageId":"2f5853a7fae7967ba959ae16c81c90b6be52a536.1715024899.git.steadmon@google.com","threadId":"60754","inReplyTo":"cover.1715024899.git.steadmon@google.com","subject":"[PATCH v6 7/7] ci: use test-tool as unit test runner on Windows","fromName":"Josh Steadmon","fromEmail":"steadmon@google.com","sentAt":"2024-05-06T19:57:37Z","receivedAt":"2024-05-06T19:57:52Z","isPatch":true,"sender":{"key":"steadmon@google.com","avatar":"https://avatars.githubusercontent.com/u/2654920?v=4"},"body":"Although the previous commit changed t/Makefile to run unit tests\nalongside shell tests, the Windows CI still needs a separate unit-tests\nstep due to how the test sharding works.\n\nWe want to avoid using `prove` as a test running on Windows due to\nperformance issues [1], so use the new test-tool runner instead.\n\n[1] https://lore.kernel.org/git/850ea42c-f103-68d5-896b-9120e2628686@gmx.de/\n\nSigned-off-by: Josh Steadmon \n---\n ci/run-test-slice.sh | 2 +-\n 1 file changed, 1 insertion(+), 1 deletion(-)\n\ndiff --git a/ci/run-test-slice.sh b/ci/run-test-slice.sh\nindex ae8094382f..e167e646f7 100755\n--- a/ci/run-test-slice.sh\n+++ b/ci/run-test-slice.sh\n@@ -17,7 +17,7 @@ handle_failed_tests\n \n # We only have one unit test at the moment, so run it in the first slice\n if [ \"$1\" == \"0\" ] ; then\n-\tgroup \"Run unit tests\" make --quiet -C t unit-tests-prove\n+\tgroup \"Run unit tests\" make --quiet -C t unit-tests-test-tool\n fi\n \n check_unignored_build_artifacts\n-- \n2.45.0.rc1.225.g2a3ae87e7f-goog\n\n"},{"id":"494195","messageId":"pqtplukr2sni4vhgxjfr5u3bsoinjukopggktzgfvdzpceibpm@judi4dfpaz3h","threadId":"60754","inReplyTo":"xmqqle4uy328.fsf@gitster.g","subject":"Re: [PATCH v5 6/7] t/Makefile: run unit tests alongside shell tests","fromName":"Josh Steadmon","fromEmail":"steadmon@google.com","sentAt":"2024-05-06T19:58:09Z","receivedAt":"2024-05-06T19:58:15Z","isPatch":true,"sender":{"key":"steadmon@google.com","avatar":"https://avatars.githubusercontent.com/u/2654920?v=4"},"body":"On 2024.04.30 14:05, Junio C Hamano wrote:\n> Josh Steadmon writes:\n> \n> > +# A simple wrapper to run shell tests via TEST_SHELL_PATH,\n> > +# or exec unit tests directly.\n> > +\n> > +case \"$1\" in\n> > +*.sh)\n> > +\tif test -z \"${TEST_SHELL_PATH+set}\"\n> > +\tthen\n> > +\t\techo >&2 \"ERROR: TEST_SHELL_PATH is empty or not set\"\n> > +\t\texit 1\n> > +\tfi\n> > +\texec ${TEST_SHELL_PATH} \"$@\"\n> \n> You want to be prepared for a path like \"//C/My Programs/bash\".\n> \n> \texec \"$TEST_SHELL_PATH\" \"$@\"\n> \n> > +\t;;\n> > +*)\n> > +\texec \"$@\"\n> > +\t;;\n> > +esac\n\nFixed in V6.\n"},{"id":"494196","messageId":"z3k5phrp5cbxyorie3fhkvhewltz5z7lqumgraxiieur3gm5qr@ei753rpieyyf","threadId":"60754","inReplyTo":"20240503180239.GA3634479@coredump.intra.peff.net","subject":"Re: [PATCH v4 6/7] t/Makefile: run unit tests alongside shell tests","fromName":"Josh Steadmon","fromEmail":"steadmon@google.com","sentAt":"2024-05-06T19:58:35Z","receivedAt":"2024-05-06T19:58:41Z","isPatch":true,"sender":{"key":"steadmon@google.com","avatar":"https://avatars.githubusercontent.com/u/2654920?v=4"},"body":"On 2024.05.03 14:02, Jeff King wrote:\n> On Wed, Apr 24, 2024 at 02:25:44PM -0700, Junio C Hamano wrote:\n> \n> > Josh Steadmon writes:\n> > \n> > > +case \"$1\" in\n> > > +*.sh)\n> > > +\tif test -z \"${TEST_SHELL_PATH+set}\" ; then\n> > > +\t\techo \"ERROR: TEST_SHELL_PATH is not set\" >&2\n> > \n> > Style.\n> > \n> > As an empty string is not a reasonable value for this variable (and\n> > you do not quote ${TEST_SHELL_PATH} when you use it in \"exec\" below),\n> > \n> > \tif test -z \"${TEST_SHELL_PATH:+set}\"\n> > \tthen\n> > \t\techo >&2 \"ERROR: TEST_SHELL_PATH is not set or empty\"\n> > \n> > may be what we want here.\n> \n> If we are using \":+\" to handle the empty string, I think just:\n> \n> if test -z \"$TEST_SHELL_PATH\"\n> \n> is sufficient, no?\n> \n> (not that the other is incorrect, but whenever I see something like\n> \":+set\" I wonder if something more clever is going on, and of course I\n> get nightmare flashbacks to looking at generated autoconf code).\n> \n> -Peff\n\nFixed in V6.\n"},{"id":"494197","messageId":"agz4p6ilfrlxnt3wjjxiqpecosne5ipax32fskzgh722xxryzz@7ddif3kdujkt","threadId":"60754","inReplyTo":"xmqqbk5qy2l3.fsf@gitster.g","subject":"Re: [PATCH v5 0/7] test-tool: add unit test suite runner","fromName":"Josh Steadmon","fromEmail":"steadmon@google.com","sentAt":"2024-05-06T20:02:07Z","receivedAt":"2024-05-06T20:02:12Z","isPatch":true,"sender":{"key":"steadmon@google.com","avatar":"https://avatars.githubusercontent.com/u/2654920?v=4"},"body":"On 2024.04.30 14:15, Junio C Hamano wrote:\n> Josh Steadmon writes:\n> \n> > For various reasons (see discussion at [1]) we would like an alternative\n> > to `prove` for running test suites (including the unit tests) on\n> > Windows.\n> >\n> > [1] https://lore.kernel.org/git/pull.1613.git.1699894837844.gitgitgadget@gmail.com/\n> >\n> > This series extends the existing `test-tool run-command testsuite` to\n> > support running unit tests. In addition, it includes some small\n> > cleanups:\n> > * move t-basic out of the unit-tests directory\n> > * don't hardcode the shell for running tests in `test-tool ... testsuite`\n> > * don't hardcode a test name filter in `test-tool ... testsuite`\n> > * add a test wrapper script to allow unit tests and the shell test suite\n> > to run in a single `prove` process\n> \n> I am OK to see it outside the scope of this series, but we would\n> need unit tests supported by the GIT_SKIP_TESTS mechanism (or an\n> alternative mechanism written), given that I hear \"migrate tests to\n> unit-test framework\" every once in a while, which means we would\n> accumlate more and more tests that ignore GIT_SKIP_TESTS mechansim.\n> \n> I did spot one potential problem (rather, \"we'd want to fix it as we\n> are changing it with this reroll anyway\"), but other than that I did\n> not see anything wrong in the other patches.\n> \n> Thanks. Will queue.\n\nAck. I think the current GIT_SKIP_TESTS system should be fine. I'll put\nit on my TODO but can't promise that it will be a priority yet.\n"},{"id":"494204","messageId":"xmqqedaey7bo.fsf@gitster.g","threadId":"60754","inReplyTo":"cover.1715024899.git.steadmon@google.com","subject":"Re: [PATCH v6 0/7] test-tool: add unit test suite runner","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2024-05-06T21:11:07Z","receivedAt":"2024-05-06T21:11:10Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Josh Steadmon writes:\n\n> For various reasons (see discussion at [1]) we would like an alternative\n> to `prove` for running test suites (including the unit tests) on\n> Windows.\n\nAs I got tired of keeping that \"Windows\" bit outside this series as\na separate and dependent topic, let me see if I can get rid of\njs/cmake-with-test-tool topic by queuing it at the tip of these 7\npatches, making it an 8-patch series. It looks like these 7 patches\nare pretty well done, so let's plan to merge them together down to\n'next' soonish.\n\nThanks.\n"}]}