{"thread":{"id":"45220","subject":"[PATCH 0/2] interoperability test harness","startedAt":"2017-02-25T09:39:20Z","lastAt":"2017-02-25T10:04:29Z","messageCount":3,"participants":["Jeff King"],"isPatch":true,"patchVersion":1,"patchTotal":2},"messages":[{"id":"312624","messageId":"20170225093231.k7jtvx47jieka7qm@sigill.intra.peff.net","threadId":"45220","inReplyTo":null,"subject":"[PATCH 0/2] interoperability test harness","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2017-02-25T09:32:32Z","receivedAt":"2017-02-25T09:39:20Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"This series adds a small test harness for interoperability tests. The\nheavy lifting is done by the normal test-lib.sh; this just makes it easy\nfor you to have access to two git versions at the same time.\n\nThis is something I've wanted a few times in the past when we make a\nfix that can only be tested when interacting with a different version of\ngit.  As we start to work on changes like new protocols or hash\nfunctions, this will hopefully make it easier to demonstrate what\nhappens when an older version of git encounters our new features.\n\nThis doesn't run when the regular test suite runs (it's likely to be a\nbit flakier, as it actually has to build the alternative versions\nseparately). So I don't necessarily expect people to run it all the\ntime. But it lets us write down in a repeatable way the sorts of testing\nthat often ends up being done manually (or not at all) today.\n\nThis series just adds the harness and a basic test that we can still\nclone from modern git using v1.0.0 (yay!). If people are interested, I\nsuspect there are previous cases that could be backfilled. A few I can\nthink of are:\n\n  1. How older versions handle repositoryformatversion=2.\n\n  2. Newer clients hitting older servers without various capabilities.\n     One example is in:\n\n       http://public-inbox.org/git/1433961320-1366-1-git-send-email-adgar@google.com/\n\n  3. Vice-versa: older clients without capabilities hitting newer\n     servers (especially in exotic situations, like shallow clone).\n\nI don't think there's a huge value in doing that for old changes unless\nsomebody has actively reported a problem. So only do it if it sounds\nlike a fun experiment. :)\n\n  [1/2]: t: add an interoperability test harness\n  [2/2]: t/interop: add test of old clients against modern git-daemon\n\n Makefile                      |  3 ++\n t/interop/.gitignore          |  4 ++\n t/interop/Makefile            | 16 ++++++++\n t/interop/README              | 84 +++++++++++++++++++++++++++++++++++++++\n t/interop/i0000-basic.sh      | 27 +++++++++++++\n t/interop/i5500-git-daemon.sh | 41 +++++++++++++++++++\n t/interop/interop-lib.sh      | 92 +++++++++++++++++++++++++++++++++++++++++++\n t/lib-git-daemon.sh           |  3 +-\n 8 files changed, 269 insertions(+), 1 deletion(-)\n create mode 100644 t/interop/.gitignore\n create mode 100644 t/interop/Makefile\n create mode 100644 t/interop/README\n create mode 100755 t/interop/i0000-basic.sh\n create mode 100755 t/interop/i5500-git-daemon.sh\n create mode 100644 t/interop/interop-lib.sh\n\n-Peff\n"},{"id":"312625","messageId":"20170225093707.egv3vc7szn57rhy4@sigill.intra.peff.net","threadId":"45220","inReplyTo":"20170225093231.k7jtvx47jieka7qm@sigill.intra.peff.net","subject":"[PATCH 1/2] t: add an interoperability test harness","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2017-02-25T09:37:07Z","receivedAt":"2017-02-25T09:45:01Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"The current test suite is good at letting you test a\nparticular version of Git. But it's not very good at letting\nyou test _two_ versions and seeing how they interact (e.g.,\none cloning from the other).\n\nThis commit adds a test harness that will build two\narbitrary versions of git and make it easy to call them from\ninside your tests. See the README and the example script for\ndetails.\n\nSigned-off-by: Jeff King <peff@peff.net>\n---\n Makefile                 |  3 ++\n t/interop/.gitignore     |  4 +++\n t/interop/Makefile       | 16 +++++++++\n t/interop/README         | 85 ++++++++++++++++++++++++++++++++++++++++++++\n t/interop/i0000-basic.sh | 27 ++++++++++++++\n t/interop/interop-lib.sh | 92 ++++++++++++++++++++++++++++++++++++++++++++++++\n 6 files changed, 227 insertions(+)\n create mode 100644 t/interop/.gitignore\n create mode 100644 t/interop/Makefile\n create mode 100644 t/interop/README\n create mode 100755 t/interop/i0000-basic.sh\n create mode 100644 t/interop/interop-lib.sh\n\ndiff --git a/Makefile b/Makefile\nindex 8e4081e06..7156b051e 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -2254,6 +2254,9 @@ endif\n ifdef GIT_PERF_MAKE_OPTS\n \t@echo GIT_PERF_MAKE_OPTS=\\''$(subst ','\\'',$(subst ','\\'',$(GIT_PERF_MAKE_OPTS)))'\\' >>$@+\n endif\n+ifdef GIT_INTEROP_MAKE_OPTS\n+\t@echo GIT_INTEROP_MAKE_OPTS=\\''$(subst ','\\'',$(subst ','\\'',$(GIT_INTEROP_MAKE_OPTS)))'\\' >>$@+\n+endif\n ifdef TEST_GIT_INDEX_VERSION\n \t@echo TEST_GIT_INDEX_VERSION=\\''$(subst ','\\'',$(subst ','\\'',$(TEST_GIT_INDEX_VERSION)))'\\' >>$@+\n endif\ndiff --git a/t/interop/.gitignore b/t/interop/.gitignore\nnew file mode 100644\nindex 000000000..49c78d3db\n--- /dev/null\n+++ b/t/interop/.gitignore\n@@ -0,0 +1,4 @@\n+/trash directory*/\n+/test-results/\n+/.prove/\n+/build/\ndiff --git a/t/interop/Makefile b/t/interop/Makefile\nnew file mode 100644\nindex 000000000..31a4bbc71\n--- /dev/null\n+++ b/t/interop/Makefile\n@@ -0,0 +1,16 @@\n+-include ../../config.mak\n+export GIT_TEST_OPTIONS\n+\n+SHELL_PATH ?= $(SHELL)\n+SHELL_PATH_SQ = $(subst ','\\'',$(SHELL_PATH))\n+T = $(sort $(wildcard i[0-9][0-9][0-9][0-9]-*.sh))\n+\n+all: $(T)\n+\n+$(T):\n+\t@echo \"*** $@ ***\"; '$(SHELL_PATH_SQ)' $@ $(GIT_TEST_OPTS)\n+\n+clean:\n+\trm -rf build \"trash directory\".* test-results\n+\n+.PHONY: all clean $(T)\ndiff --git a/t/interop/README b/t/interop/README\nnew file mode 100644\nindex 000000000..72d42bd85\n--- /dev/null\n+++ b/t/interop/README\n@@ -0,0 +1,85 @@\n+Git version interoperability tests\n+==================================\n+\n+This directory has interoperability tests for git. Each script is\n+similar to the normal test scripts found in t/, but with the added twist\n+that two special versions of git, \"git.a\" and \"git.b\", are available in\n+the PATH. Individual tests can then check the interaction between the\n+two versions.\n+\n+When you add a feature that handles backwards compatibility between git\n+versions, it's encouraged to add a test here to make sure it behaves as\n+you expect.\n+\n+\n+Running Tests\n+-------------\n+\n+The easiest way to run tests is to say \"make\".  This runs all\n+the tests against their default versions.\n+\n+You can run a single test like:\n+\n+    $ ./i0000-basic.sh\n+    ok 1 - bare git is forbidden\n+    ok 2 - git.a version (v1.6.6.3)\n+    ok 3 - git.b version (v2.11.1)\n+    # passed all 3 test(s)\n+    1..3\n+\n+Each test contains default versions to run against. You may override\n+these by setting `GIT_TEST_VERSION_A` and `GIT_TEST_VERSION_B` in the\n+environment. Note that not all combinations will give sensible outcomes\n+for all tests (e.g., a test checking for a specific old/new interaction\n+may want something \"old\" enough\" and something \"new\" enough; see\n+individual tests for details).\n+\n+Version names should be resolvable as revisions in the current\n+repository. They will be exported and built as needed using the\n+config.mak files found at the root of your working tree.\n+\n+The exception is the special version \".\" which uses the currently-built\n+contents of your working tree.\n+\n+You can set the following variables (in the environment or in your config.mak):\n+\n+    GIT_INTEROP_MAKE_OPTS\n+\tOptions to pass to `make` when building a git version (e.g.,\n+\t`-j8`).\n+\n+You can also pass any command-line options taken by ordinary git tests (e.g.,\n+\"-v\").\n+\n+\n+Naming Tests\n+------------\n+\n+The interop test files are named like:\n+\n+\tiNNNN-short-description.sh\n+\n+where N is a decimal digit.  The same conventions for choosing NNNN as\n+for normal tests apply.\n+\n+\n+Writing Tests\n+-------------\n+\n+An interop test script starts like a normal script, declaring a few\n+variables and then including interop-lib.sh (which includes test-lib.sh).\n+Besides test_description, you should also set the $VERSION_A and $VERSION_B\n+variables to give the default versions to test against. See t0000-basic.sh for\n+an example.\n+\n+You can then use test_expect_success as usual, with a few differences:\n+\n+  1. The special commands \"git.a\" and \"git.b\" correspond to the\n+     two versions.\n+\n+  2. You cannot call a bare \"git\". This is to prevent accidents where\n+     you meant \"git.a\" or \"git.b\".\n+\n+  3. The trash directory is _not_ a git repository by default. You\n+     should create one with the appropriate version of git.\n+\n+At the end of the script, call test_done as usual.\ndiff --git a/t/interop/i0000-basic.sh b/t/interop/i0000-basic.sh\nnew file mode 100755\nindex 000000000..903e9193f\n--- /dev/null\n+++ b/t/interop/i0000-basic.sh\n@@ -0,0 +1,27 @@\n+#!/bin/sh\n+\n+# Note that this test only works on real version numbers,\n+# as it depends on matching the output to \"git version\".\n+VERSION_A=v1.6.6.3\n+VERSION_B=v2.11.1\n+\n+test_description='sanity test interop library'\n+. ./interop-lib.sh\n+\n+test_expect_success 'bare git is forbidden' '\n+\ttest_must_fail git version\n+'\n+\n+test_expect_success \"git.a version ($VERSION_A)\" '\n+\techo git version ${VERSION_A#v} >expect &&\n+\tgit.a version >actual &&\n+\ttest_cmp expect actual\n+'\n+\n+test_expect_success \"git.b version ($VERSION_B)\" '\n+\techo git version ${VERSION_B#v} >expect &&\n+\tgit.b version >actual &&\n+\ttest_cmp expect actual\n+'\n+\n+test_done\ndiff --git a/t/interop/interop-lib.sh b/t/interop/interop-lib.sh\nnew file mode 100644\nindex 000000000..7f31d9cc6\n--- /dev/null\n+++ b/t/interop/interop-lib.sh\n@@ -0,0 +1,92 @@\n+# Interoperability testing framework. Each script should source\n+# this after setting default $VERSION_A and $VERSION_B variables.\n+\n+. ../../GIT-BUILD-OPTIONS\n+INTEROP_ROOT=$(pwd)\n+BUILD_ROOT=$INTEROP_ROOT/build\n+\n+build_version() {\n+\tif test -z \"$1\"\n+\tthen\n+\t\techo >&2 \"error: test script did not set default versions\"\n+\t\treturn 1\n+\tfi\n+\n+\tif test \"$1\" = \".\"\n+\tthen\n+\t\tgit rev-parse --show-toplevel\n+\t\treturn 0\n+\tfi\n+\n+\tsha1=$(git rev-parse \"$1^{tree}\") || return 1\n+\tdir=$BUILD_ROOT/$sha1\n+\n+\tif test -e \"$dir/.built\"\n+\tthen\n+\t\techo \"$dir\"\n+\t\treturn 0\n+\tfi\n+\n+\techo >&2 \"==> Building $1...\"\n+\n+\tmkdir -p \"$dir\" || return 1\n+\n+\t(cd \"$(git rev-parse --show-cdup)\" && git archive --format=tar \"$sha1\") |\n+\t(cd \"$dir\" && tar x) ||\n+\treturn 1\n+\n+\tfor config in config.mak config.mak.autogen config.status\n+\tdo\n+\t\tif test -e \"$INTEROP_ROOT/../../$config\"\n+\t\tthen\n+\t\t\tcp \"$INTEROP_ROOT/../../$config\" \"$dir/\" || return 1\n+\t\tfi\n+\tdone\n+\n+\t(\n+\t\tcd \"$dir\" &&\n+\t\tmake $GIT_INTEROP_MAKE_OPTS >&2 &&\n+\t\ttouch .built\n+\t) || return 1\n+\n+\techo \"$dir\"\n+}\n+\n+# Old versions of git don't have bin-wrappers, so let's give a rough emulation.\n+wrap_git() {\n+\twrite_script \"$1\" <<-EOF\n+\tGIT_EXEC_PATH=\"$2\"\n+\texport GIT_EXEC_PATH\n+\tPATH=\"$2:\\$PATH\"\n+\texport GIT_EXEC_PATH\n+\texec git \"\\$@\"\n+\tEOF\n+}\n+\n+generate_wrappers() {\n+\tmkdir -p .bin &&\n+\twrap_git .bin/git.a \"$DIR_A\" &&\n+\twrap_git .bin/git.b \"$DIR_B\" &&\n+\twrite_script .bin/git <<-\\EOF &&\n+\techo >&2 fatal: test tried to run generic git\n+\texit 1\n+\tEOF\n+\tPATH=$(pwd)/.bin:$PATH\n+}\n+\n+VERSION_A=${GIT_TEST_VERSION_A:-$VERSION_A}\n+VERSION_B=${GIT_TEST_VERSION_B:-$VERSION_B}\n+\n+if ! DIR_A=$(build_version \"$VERSION_A\") ||\n+   ! DIR_B=$(build_version \"$VERSION_B\")\n+then\n+\techo >&2 \"fatal: unable to build git versions\"\n+\texit 1\n+fi\n+\n+TEST_DIRECTORY=$INTEROP_ROOT/..\n+TEST_OUTPUT_DIRECTORY=$INTEROP_ROOT\n+TEST_NO_CREATE_REPO=t\n+. \"$TEST_DIRECTORY\"/test-lib.sh\n+\n+generate_wrappers || die \"unable to set up interop test environment\"\n-- \n2.12.0.616.g5f622f3b1\n\n"},{"id":"312627","messageId":"20170225093730.phcefrsjzcr3jwvr@sigill.intra.peff.net","threadId":"45220","inReplyTo":"20170225093231.k7jtvx47jieka7qm@sigill.intra.peff.net","subject":"[PATCH 2/2] t/interop: add test of old clients against modern git-daemon","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2017-02-25T09:37:30Z","receivedAt":"2017-02-25T10:04:29Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"This test just checks that old clients can clone and fetch\nfrom a newer git-daemon. The opposite should also be true,\nbut it's hard to test ancient versions of git-daemon because\nthey lack basic options like \"--listen\".\n\nNote that we have to make a slight tweak to the\nlib-git-daemon helper from the regular tests, so that it\nstarts the daemon with our correct git.a version.\n\nSigned-off-by: Jeff King <peff@peff.net>\n---\n t/interop/i5500-git-daemon.sh | 41 +++++++++++++++++++++++++++++++++++++++++\n t/lib-git-daemon.sh           |  3 ++-\n 2 files changed, 43 insertions(+), 1 deletion(-)\n create mode 100755 t/interop/i5500-git-daemon.sh\n\ndiff --git a/t/interop/i5500-git-daemon.sh b/t/interop/i5500-git-daemon.sh\nnew file mode 100755\nindex 000000000..1daf69420\n--- /dev/null\n+++ b/t/interop/i5500-git-daemon.sh\n@@ -0,0 +1,41 @@\n+#!/bin/sh\n+\n+VERSION_A=.\n+VERSION_B=v1.0.0\n+\n+: ${LIB_GIT_DAEMON_PORT:=5500}\n+LIB_GIT_DAEMON_COMMAND='git.a daemon'\n+\n+test_description='clone and fetch by older client'\n+. ./interop-lib.sh\n+. \"$TEST_DIRECTORY\"/lib-git-daemon.sh\n+\n+start_git_daemon --export-all\n+\n+repo=$GIT_DAEMON_DOCUMENT_ROOT_PATH/repo\n+\n+test_expect_success \"create repo served by $VERSION_A\" '\n+\tgit.a init \"$repo\" &&\n+\tgit.a -C \"$repo\" commit --allow-empty -m one\n+'\n+\n+test_expect_success \"clone with $VERSION_B\" '\n+\tgit.b clone \"$GIT_DAEMON_URL/repo\" child &&\n+\techo one >expect &&\n+\tgit.a -C child log -1 --format=%s >actual &&\n+\ttest_cmp expect actual\n+'\n+\n+test_expect_success \"fetch with $VERSION_B\" '\n+\tgit.a -C \"$repo\" commit --allow-empty -m two &&\n+\t(\n+\t\tcd child &&\n+\t\tgit.b fetch\n+\t) &&\n+\techo two >expect &&\n+\tgit.a -C child log -1 --format=%s FETCH_HEAD >actual &&\n+\ttest_cmp expect actual\n+'\n+\n+stop_git_daemon\n+test_done\ndiff --git a/t/lib-git-daemon.sh b/t/lib-git-daemon.sh\nindex f9cbd4793..987d40680 100644\n--- a/t/lib-git-daemon.sh\n+++ b/t/lib-git-daemon.sh\n@@ -46,7 +46,8 @@ start_git_daemon() {\n \n \tsay >&3 \"Starting git daemon ...\"\n \tmkfifo git_daemon_output\n-\tgit daemon --listen=127.0.0.1 --port=\"$LIB_GIT_DAEMON_PORT\" \\\n+\t${LIB_GIT_DAEMON_COMMAND:-git daemon} \\\n+\t\t--listen=127.0.0.1 --port=\"$LIB_GIT_DAEMON_PORT\" \\\n \t\t--reuseaddr --verbose \\\n \t\t--base-path=\"$GIT_DAEMON_DOCUMENT_ROOT_PATH\" \\\n \t\t\"$@\" \"$GIT_DAEMON_DOCUMENT_ROOT_PATH\" \\\n-- \n2.12.0.616.g5f622f3b1\n"}]}