threads / patch / 45220

patch, 2 partsinteroperability test harness

Subject: [PATCH 0/2] interoperability test harness

## tl;dr

3 messages between Feb 25, 2017 and Feb 25, 2017. Diffs are folded; open one to read it.

replies: 2people: 1as markdown or json

Jeff King· Feb 25, 2017, 09:32 UTC · lore

This series adds a small test harness for interoperability tests. The heavy lifting is done by the normal test-lib.sh; this just makes it easy for you to have access to two git versions at the same time.

This is something I've wanted a few times in the past when we make a fix that can only be tested when interacting with a different version of git. As we start to work on changes like new protocols or hash functions, this will hopefully make it easier to demonstrate what happens when an older version of git encounters our new features.

This doesn't run when the regular test suite runs (it's likely to be a bit flakier, as it actually has to build the alternative versions separately). So I don't necessarily expect people to run it all the time. But it lets us write down in a repeatable way the sorts of testing that often ends up being done manually (or not at all) today.

This series just adds the harness and a basic test that we can still clone from modern git using v1.0.0 (yay!). If people are interested, I suspect there are previous cases that could be backfilled. A few I can think of are:

  1. How older versions handle repositoryformatversion=2.
  2. Newer clients hitting older servers without various capabilities.
     One example is in:
       http://public-inbox.org/git/1433961320-1366-1-git-send-email-adgar@google.com/
  3. Vice-versa: older clients without capabilities hitting newer
     servers (especially in exotic situations, like shallow clone).

I don't think there's a huge value in doing that for old changes unless somebody has actively reported a problem. So only do it if it sounds like a fun experiment. :)

  [1/2]: t: add an interoperability test harness
  [2/2]: t/interop: add test of old clients against modern git-daemon
 Makefile                      |  3 ++
 t/interop/.gitignore          |  4 ++
 t/interop/Makefile            | 16 ++++++++
 t/interop/README              | 84 +++++++++++++++++++++++++++++++++++++++
 t/interop/i0000-basic.sh      | 27 +++++++++++++
 t/interop/i5500-git-daemon.sh | 41 +++++++++++++++++++
 t/interop/interop-lib.sh      | 92 +++++++++++++++++++++++++++++++++++++++++++
 t/lib-git-daemon.sh           |  3 +-
 8 files changed, 269 insertions(+), 1 deletion(-)
 create mode 100644 t/interop/.gitignore
 create mode 100644 t/interop/Makefile
 create mode 100644 t/interop/README
 create mode 100755 t/interop/i0000-basic.sh
 create mode 100755 t/interop/i5500-git-daemon.sh
 create mode 100644 t/interop/interop-lib.sh
-Peff
Jeff King· Feb 25, 2017, 09:37 UTC · re: Jeff King · lore

[PATCH 1/2] t: add an interoperability test harness

The current test suite is good at letting you test a particular version of Git. But it's not very good at letting you test _two_ versions and seeing how they interact (e.g., one cloning from the other).

This commit adds a test harness that will build two arbitrary versions of git and make it easy to call them from inside your tests. See the README and the example script for details.

Signed-off-by: Jeff King <peff@peff.net>
---
 Makefile                 |  3 ++
 t/interop/.gitignore     |  4 +++
 t/interop/Makefile       | 16 +++++++++
 t/interop/README         | 85 ++++++++++++++++++++++++++++++++++++++++++++
 t/interop/i0000-basic.sh | 27 ++++++++++++++
 t/interop/interop-lib.sh | 92 ++++++++++++++++++++++++++++++++++++++++++++++++
 6 files changed, 227 insertions(+)
 create mode 100644 t/interop/.gitignore
 create mode 100644 t/interop/Makefile
 create mode 100644 t/interop/README
 create mode 100755 t/interop/i0000-basic.sh
 create mode 100644 t/interop/interop-lib.sh
Show changes to 6 files +227 −0

Makefile, t/interop/.gitignore, t/interop/Makefile, t/interop/README, t/interop/i0000-basic.sh, t/interop/interop-lib.sh

diff --git a/Makefile b/Makefile
index 8e4081e06..7156b051e 100644
--- a/Makefile
+++ b/Makefile
@@ -2254,6 +2254,9 @@ endif
 ifdef GIT_PERF_MAKE_OPTS
 	@echo GIT_PERF_MAKE_OPTS=\''$(subst ','\'',$(subst ','\'',$(GIT_PERF_MAKE_OPTS)))'\' >>$@+
 endif
+ifdef GIT_INTEROP_MAKE_OPTS
+	@echo GIT_INTEROP_MAKE_OPTS=\''$(subst ','\'',$(subst ','\'',$(GIT_INTEROP_MAKE_OPTS)))'\' >>$@+
+endif
 ifdef TEST_GIT_INDEX_VERSION
 	@echo TEST_GIT_INDEX_VERSION=\''$(subst ','\'',$(subst ','\'',$(TEST_GIT_INDEX_VERSION)))'\' >>$@+
 endif
diff --git a/t/interop/.gitignore b/t/interop/.gitignore
new file mode 100644
index 000000000..49c78d3db
--- /dev/null
+++ b/t/interop/.gitignore
@@ -0,0 +1,4 @@
+/trash directory*/
+/test-results/
+/.prove/
+/build/
diff --git a/t/interop/Makefile b/t/interop/Makefile
new file mode 100644
index 000000000..31a4bbc71
--- /dev/null
+++ b/t/interop/Makefile
@@ -0,0 +1,16 @@
+-include ../../config.mak
+export GIT_TEST_OPTIONS
+
+SHELL_PATH ?= $(SHELL)
+SHELL_PATH_SQ = $(subst ','\'',$(SHELL_PATH))
+T = $(sort $(wildcard i[0-9][0-9][0-9][0-9]-*.sh))
+
+all: $(T)
+
+$(T):
+	@echo "*** $@ ***"; '$(SHELL_PATH_SQ)' $@ $(GIT_TEST_OPTS)
+
+clean:
+	rm -rf build "trash directory".* test-results
+
+.PHONY: all clean $(T)
diff --git a/t/interop/README b/t/interop/README
new file mode 100644
index 000000000..72d42bd85
--- /dev/null
+++ b/t/interop/README
@@ -0,0 +1,85 @@
+Git version interoperability tests
+==================================
+
+This directory has interoperability tests for git. Each script is
+similar to the normal test scripts found in t/, but with the added twist
+that two special versions of git, "git.a" and "git.b", are available in
+the PATH. Individual tests can then check the interaction between the
+two versions.
+
+When you add a feature that handles backwards compatibility between git
+versions, it's encouraged to add a test here to make sure it behaves as
+you expect.
+
+
+Running Tests
+-------------
+
+The easiest way to run tests is to say "make".  This runs all
+the tests against their default versions.
+
+You can run a single test like:
+
+    $ ./i0000-basic.sh
+    ok 1 - bare git is forbidden
+    ok 2 - git.a version (v1.6.6.3)
+    ok 3 - git.b version (v2.11.1)
+    # passed all 3 test(s)
+    1..3
+
+Each test contains default versions to run against. You may override
+these by setting `GIT_TEST_VERSION_A` and `GIT_TEST_VERSION_B` in the
+environment. Note that not all combinations will give sensible outcomes
+for all tests (e.g., a test checking for a specific old/new interaction
+may want something "old" enough" and something "new" enough; see
+individual tests for details).
+
+Version names should be resolvable as revisions in the current
+repository. They will be exported and built as needed using the
+config.mak files found at the root of your working tree.
+
+The exception is the special version "." which uses the currently-built
+contents of your working tree.
+
+You can set the following variables (in the environment or in your config.mak):
+
+    GIT_INTEROP_MAKE_OPTS
+	Options to pass to `make` when building a git version (e.g.,
+	`-j8`).
+
+You can also pass any command-line options taken by ordinary git tests (e.g.,
+"-v").
+
+
+Naming Tests
+------------
+
+The interop test files are named like:
+
+	iNNNN-short-description.sh
+
+where N is a decimal digit.  The same conventions for choosing NNNN as
+for normal tests apply.
+
+
+Writing Tests
+-------------
+
+An interop test script starts like a normal script, declaring a few
+variables and then including interop-lib.sh (which includes test-lib.sh).
+Besides test_description, you should also set the $VERSION_A and $VERSION_B
+variables to give the default versions to test against. See t0000-basic.sh for
+an example.
+
+You can then use test_expect_success as usual, with a few differences:
+
+  1. The special commands "git.a" and "git.b" correspond to the
+     two versions.
+
+  2. You cannot call a bare "git". This is to prevent accidents where
+     you meant "git.a" or "git.b".
+
+  3. The trash directory is _not_ a git repository by default. You
+     should create one with the appropriate version of git.
+
+At the end of the script, call test_done as usual.
diff --git a/t/interop/i0000-basic.sh b/t/interop/i0000-basic.sh
new file mode 100755
index 000000000..903e9193f
--- /dev/null
+++ b/t/interop/i0000-basic.sh
@@ -0,0 +1,27 @@
+#!/bin/sh
+
+# Note that this test only works on real version numbers,
+# as it depends on matching the output to "git version".
+VERSION_A=v1.6.6.3
+VERSION_B=v2.11.1
+
+test_description='sanity test interop library'
+. ./interop-lib.sh
+
+test_expect_success 'bare git is forbidden' '
+	test_must_fail git version
+'
+
+test_expect_success "git.a version ($VERSION_A)" '
+	echo git version ${VERSION_A#v} >expect &&
+	git.a version >actual &&
+	test_cmp expect actual
+'
+
+test_expect_success "git.b version ($VERSION_B)" '
+	echo git version ${VERSION_B#v} >expect &&
+	git.b version >actual &&
+	test_cmp expect actual
+'
+
+test_done
diff --git a/t/interop/interop-lib.sh b/t/interop/interop-lib.sh
new file mode 100644
index 000000000..7f31d9cc6
--- /dev/null
+++ b/t/interop/interop-lib.sh
@@ -0,0 +1,92 @@
+# Interoperability testing framework. Each script should source
+# this after setting default $VERSION_A and $VERSION_B variables.
+
+. ../../GIT-BUILD-OPTIONS
+INTEROP_ROOT=$(pwd)
+BUILD_ROOT=$INTEROP_ROOT/build
+
+build_version() {
+	if test -z "$1"
+	then
+		echo >&2 "error: test script did not set default versions"
+		return 1
+	fi
+
+	if test "$1" = "."
+	then
+		git rev-parse --show-toplevel
+		return 0
+	fi
+
+	sha1=$(git rev-parse "$1^{tree}") || return 1
+	dir=$BUILD_ROOT/$sha1
+
+	if test -e "$dir/.built"
+	then
+		echo "$dir"
+		return 0
+	fi
+
+	echo >&2 "==> Building $1..."
+
+	mkdir -p "$dir" || return 1
+
+	(cd "$(git rev-parse --show-cdup)" && git archive --format=tar "$sha1") |
+	(cd "$dir" && tar x) ||
+	return 1
+
+	for config in config.mak config.mak.autogen config.status
+	do
+		if test -e "$INTEROP_ROOT/../../$config"
+		then
+			cp "$INTEROP_ROOT/../../$config" "$dir/" || return 1
+		fi
+	done
+
+	(
+		cd "$dir" &&
+		make $GIT_INTEROP_MAKE_OPTS >&2 &&
+		touch .built
+	) || return 1
+
+	echo "$dir"
+}
+
+# Old versions of git don't have bin-wrappers, so let's give a rough emulation.
+wrap_git() {
+	write_script "$1" <<-EOF
+	GIT_EXEC_PATH="$2"
+	export GIT_EXEC_PATH
+	PATH="$2:\$PATH"
+	export GIT_EXEC_PATH
+	exec git "\$@"
+	EOF
+}
+
+generate_wrappers() {
+	mkdir -p .bin &&
+	wrap_git .bin/git.a "$DIR_A" &&
+	wrap_git .bin/git.b "$DIR_B" &&
+	write_script .bin/git <<-\EOF &&
+	echo >&2 fatal: test tried to run generic git
+	exit 1
+	EOF
+	PATH=$(pwd)/.bin:$PATH
+}
+
+VERSION_A=${GIT_TEST_VERSION_A:-$VERSION_A}
+VERSION_B=${GIT_TEST_VERSION_B:-$VERSION_B}
+
+if ! DIR_A=$(build_version "$VERSION_A") ||
+   ! DIR_B=$(build_version "$VERSION_B")
+then
+	echo >&2 "fatal: unable to build git versions"
+	exit 1
+fi
+
+TEST_DIRECTORY=$INTEROP_ROOT/..
+TEST_OUTPUT_DIRECTORY=$INTEROP_ROOT
+TEST_NO_CREATE_REPO=t
+. "$TEST_DIRECTORY"/test-lib.sh
+
+generate_wrappers || die "unable to set up interop test environment"
-- 
2.12.0.616.g5f622f3b1
Jeff King· Feb 25, 2017, 09:37 UTC · re: Jeff King · lore

[PATCH 2/2] t/interop: add test of old clients against modern git-daemon

This test just checks that old clients can clone and fetch from a newer git-daemon. The opposite should also be true, but it's hard to test ancient versions of git-daemon because they lack basic options like "--listen".

Note that we have to make a slight tweak to the lib-git-daemon helper from the regular tests, so that it starts the daemon with our correct git.a version.

Signed-off-by: Jeff King <peff@peff.net>
---
 t/interop/i5500-git-daemon.sh | 41 +++++++++++++++++++++++++++++++++++++++++
 t/lib-git-daemon.sh           |  3 ++-
 2 files changed, 43 insertions(+), 1 deletion(-)
 create mode 100755 t/interop/i5500-git-daemon.sh
Show changes to 2 files +43 −1

t/interop/i5500-git-daemon.sh, t/lib-git-daemon.sh

diff --git a/t/interop/i5500-git-daemon.sh b/t/interop/i5500-git-daemon.sh
new file mode 100755
index 000000000..1daf69420
--- /dev/null
+++ b/t/interop/i5500-git-daemon.sh
@@ -0,0 +1,41 @@
+#!/bin/sh
+
+VERSION_A=.
+VERSION_B=v1.0.0
+
+: ${LIB_GIT_DAEMON_PORT:=5500}
+LIB_GIT_DAEMON_COMMAND='git.a daemon'
+
+test_description='clone and fetch by older client'
+. ./interop-lib.sh
+. "$TEST_DIRECTORY"/lib-git-daemon.sh
+
+start_git_daemon --export-all
+
+repo=$GIT_DAEMON_DOCUMENT_ROOT_PATH/repo
+
+test_expect_success "create repo served by $VERSION_A" '
+	git.a init "$repo" &&
+	git.a -C "$repo" commit --allow-empty -m one
+'
+
+test_expect_success "clone with $VERSION_B" '
+	git.b clone "$GIT_DAEMON_URL/repo" child &&
+	echo one >expect &&
+	git.a -C child log -1 --format=%s >actual &&
+	test_cmp expect actual
+'
+
+test_expect_success "fetch with $VERSION_B" '
+	git.a -C "$repo" commit --allow-empty -m two &&
+	(
+		cd child &&
+		git.b fetch
+	) &&
+	echo two >expect &&
+	git.a -C child log -1 --format=%s FETCH_HEAD >actual &&
+	test_cmp expect actual
+'
+
+stop_git_daemon
+test_done
diff --git a/t/lib-git-daemon.sh b/t/lib-git-daemon.sh
index f9cbd4793..987d40680 100644
--- a/t/lib-git-daemon.sh
+++ b/t/lib-git-daemon.sh
@@ -46,7 +46,8 @@ start_git_daemon() {
 
 	say >&3 "Starting git daemon ..."
 	mkfifo git_daemon_output
-	git daemon --listen=127.0.0.1 --port="$LIB_GIT_DAEMON_PORT" \
+	${LIB_GIT_DAEMON_COMMAND:-git daemon} \
+		--listen=127.0.0.1 --port="$LIB_GIT_DAEMON_PORT" \
 		--reuseaddr --verbose \
 		--base-path="$GIT_DAEMON_DOCUMENT_ROOT_PATH" \
 		"$@" "$GIT_DAEMON_DOCUMENT_ROOT_PATH" \
-- 
2.12.0.616.g5f622f3b1

← back to recent threads