{"thread":{"id":"29165","subject":"[RFC PATCH 0/4] Performance test framework","startedAt":"2011-12-14T15:23:28Z","lastAt":"2011-12-15T19:05:01Z","messageCount":9,"participants":["Thomas Rast","Junio C Hamano"],"isPatch":true,"patchVersion":1,"patchTotal":4},"messages":[{"id":"181145","messageId":"cover.1323876121.git.trast@student.ethz.ch","threadId":"29165","inReplyTo":null,"subject":"[RFC PATCH 0/4] Performance test framework","fromName":"Thomas Rast","fromEmail":"trast@student.ethz.ch","sentAt":"2011-12-14T15:23:28Z","receivedAt":"2011-12-14T15:23:28Z","isPatch":true,"sender":{"key":"tr@thomasrast.ch","avatar":"https://avatars.githubusercontent.com/u/153510?v=4"},"body":"Hi,\n\nThis is the first shot at a usable performance test suite.  It's\nanother angle than the refperf work of Michael Haggerty, however he\nhelped shape this in #git-devel (thanks!).\n\nThere's a big README, but if you just want to dive in, here's me\ncomparing the grep performance of my POC \"pack reading in parallel\"\nbranch against Junio's next:\n\n  $ cd t/perf\n  $ ./run origin/next t/sha1_file-parallel p7810-grep.sh \n  [snip no-op compilation]\n  === Running 1 tests in build/7a6d658aa3c9016dd04ff3515cbf15edca6562a4 ===\n  perf 1 - grep worktree, cheap regex: 1 2 3 4 5 ok\n  perf 2 - grep worktree, expensive regex: 1 2 3 4 5 ok\n  perf 3 - grep --cached, cheap regex: 1 2 3 4 5 ok\n  perf 4 - grep --cached, expensive regex: 1 2 3 4 5 ok\n  # passed all 4 test(s)\n  1..4\n  GIT_VERSION = 1.7.8.GIT\n      * new build flags or prefix\n      * new link flags\n      GEN common-cmds.h\n      CC hex.o\n      CC kwset.o\n  [snip rest of compilation]\n  === Running 1 tests in build/dd2bf650b382f5aca727b7d93a48598fb1a2f7d9 ===\n  perf 1 - grep worktree, cheap regex: 1 2 3 4 5 ok\n  perf 2 - grep worktree, expensive regex: 1 2 3 4 5 ok\n  perf 3 - grep --cached, cheap regex: 1 2 3 4 5 ok\n  perf 4 - grep --cached, expensive regex: 1 2 3 4 5 ok\n  # passed all 4 test(s)\n  1..4\n  Test                                     origin/next        t/sha1_file-parallel  \n  ----------------------------------------------------------------------------------\n  7810.1: grep worktree, cheap regex       0.16(0.16+0.35)    0.16(0.15+0.36) +0.0% \n  7810.2: grep worktree, expensive regex   7.83(29.68+0.39)   7.95(29.98+0.39) +1.5%\n  7810.3: grep --cached, cheap regex       3.12(3.11+0.24)    1.11(3.46+0.18) -64.4%\n  7810.4: grep --cached, expensive regex   9.43(30.53+0.28)   8.89(32.99+0.22) -5.7%\n\nNote in particular that neither of the two branches contains the perf\nwork.\n\nHave fun!\n\nThomas Rast (4):\n  Move the user-facing test library to test-lib-functions.sh\n  test-lib: allow testing another git build tree\n  Introduce a performance testing framework\n  Add a performance test for git-grep\n\n Makefile                        |   26 ++-\n t/Makefile                      |    5 +-\n t/perf/.gitignore               |    2 +\n t/perf/Makefile                 |   15 +\n t/perf/README                   |  146 ++++++++++\n t/perf/aggregate.perl           |  166 ++++++++++++\n t/perf/min_time.perl            |   21 ++\n t/perf/p0000-perf-lib-sanity.sh |   41 +++\n t/perf/p0001-rev-list.sh        |   17 ++\n t/perf/p7810-grep.sh            |   23 ++\n t/perf/perf-lib.sh              |  199 ++++++++++++++\n t/perf/run                      |   82 ++++++\n t/test-lib-functions.sh         |  528 ++++++++++++++++++++++++++++++++++++\n t/test-lib.sh                   |  561 +++------------------------------------\n 14 files changed, 1300 insertions(+), 532 deletions(-)\n create mode 100644 t/perf/.gitignore\n create mode 100644 t/perf/Makefile\n create mode 100644 t/perf/README\n create mode 100755 t/perf/aggregate.perl\n create mode 100755 t/perf/min_time.perl\n create mode 100755 t/perf/p0000-perf-lib-sanity.sh\n create mode 100755 t/perf/p0001-rev-list.sh\n create mode 100755 t/perf/p7810-grep.sh\n create mode 100644 t/perf/perf-lib.sh\n create mode 100755 t/perf/run\n create mode 100644 t/test-lib-functions.sh\n\n-- \n1.7.8.304.ge42e4\n"},{"id":"181147","messageId":"842cefc94e848530a43295ddf6d607f9bcbe3ab7.1323876121.git.trast@student.ethz.ch","threadId":"29165","inReplyTo":"cover.1323876121.git.trast@student.ethz.ch","subject":"[RFC PATCH 1/4] Move the user-facing test library to test-lib-functions.sh","fromName":"Thomas Rast","fromEmail":"trast@student.ethz.ch","sentAt":"2011-12-14T15:23:29Z","receivedAt":"2011-12-14T15:23:29Z","isPatch":true,"sender":{"key":"tr@thomasrast.ch","avatar":"https://avatars.githubusercontent.com/u/153510?v=4"},"body":"This just moves all the user-facing functions to a separate file and\nsources that instead.\n\nSigned-off-by: Thomas Rast <trast@student.ethz.ch>\n---\n t/test-lib-functions.sh |  528 ++++++++++++++++++++++++++++++++++++++++++++++\n t/test-lib.sh           |  530 +----------------------------------------------\n 2 files changed, 533 insertions(+), 525 deletions(-)\n create mode 100644 t/test-lib-functions.sh\n\ndiff --git a/t/test-lib-functions.sh b/t/test-lib-functions.sh\nnew file mode 100644\nindex 0000000..103c97e\n--- /dev/null\n+++ b/t/test-lib-functions.sh\n@@ -0,0 +1,528 @@\n+# The semantics of the editor variables are that of invoking\n+# sh -c \"$EDITOR \\\"$@\\\"\" files ...\n+#\n+# If our trash directory contains shell metacharacters, they will be\n+# interpreted if we just set $EDITOR directly, so do a little dance with\n+# environment variables to work around this.\n+#\n+# In particular, quoting isn't enough, as the path may contain the same quote\n+# that we're using.\n+test_set_editor () {\n+\tFAKE_EDITOR=\"$1\"\n+\texport FAKE_EDITOR\n+\tEDITOR='\"$FAKE_EDITOR\"'\n+\texport EDITOR\n+}\n+\n+test_decode_color () {\n+\tawk '\n+\t\tfunction name(n) {\n+\t\t\tif (n == 0) return \"RESET\";\n+\t\t\tif (n == 1) return \"BOLD\";\n+\t\t\tif (n == 30) return \"BLACK\";\n+\t\t\tif (n == 31) return \"RED\";\n+\t\t\tif (n == 32) return \"GREEN\";\n+\t\t\tif (n == 33) return \"YELLOW\";\n+\t\t\tif (n == 34) return \"BLUE\";\n+\t\t\tif (n == 35) return \"MAGENTA\";\n+\t\t\tif (n == 36) return \"CYAN\";\n+\t\t\tif (n == 37) return \"WHITE\";\n+\t\t\tif (n == 40) return \"BLACK\";\n+\t\t\tif (n == 41) return \"BRED\";\n+\t\t\tif (n == 42) return \"BGREEN\";\n+\t\t\tif (n == 43) return \"BYELLOW\";\n+\t\t\tif (n == 44) return \"BBLUE\";\n+\t\t\tif (n == 45) return \"BMAGENTA\";\n+\t\t\tif (n == 46) return \"BCYAN\";\n+\t\t\tif (n == 47) return \"BWHITE\";\n+\t\t}\n+\t\t{\n+\t\t\twhile (match($0, /\\033\\[[0-9;]*m/) != 0) {\n+\t\t\t\tprintf \"%s<\", substr($0, 1, RSTART-1);\n+\t\t\t\tcodes = substr($0, RSTART+2, RLENGTH-3);\n+\t\t\t\tif (length(codes) == 0)\n+\t\t\t\t\tprintf \"%s\", name(0)\n+\t\t\t\telse {\n+\t\t\t\t\tn = split(codes, ary, \";\");\n+\t\t\t\t\tsep = \"\";\n+\t\t\t\t\tfor (i = 1; i <= n; i++) {\n+\t\t\t\t\t\tprintf \"%s%s\", sep, name(ary[i]);\n+\t\t\t\t\t\tsep = \";\"\n+\t\t\t\t\t}\n+\t\t\t\t}\n+\t\t\t\tprintf \">\";\n+\t\t\t\t$0 = substr($0, RSTART + RLENGTH, length($0) - RSTART - RLENGTH + 1);\n+\t\t\t}\n+\t\t\tprint\n+\t\t}\n+\t'\n+}\n+\n+nul_to_q () {\n+\tperl -pe 'y/\\000/Q/'\n+}\n+\n+q_to_nul () {\n+\tperl -pe 'y/Q/\\000/'\n+}\n+\n+q_to_cr () {\n+\ttr Q '\\015'\n+}\n+\n+q_to_tab () {\n+\ttr Q '\\011'\n+}\n+\n+append_cr () {\n+\tsed -e 's/$/Q/' | tr Q '\\015'\n+}\n+\n+remove_cr () {\n+\ttr '\\015' Q | sed -e 's/Q$//'\n+}\n+\n+# In some bourne shell implementations, the \"unset\" builtin returns\n+# nonzero status when a variable to be unset was not set in the first\n+# place.\n+#\n+# Use sane_unset when that should not be considered an error.\n+\n+sane_unset () {\n+\tunset \"$@\"\n+\treturn 0\n+}\n+\n+test_tick () {\n+\tif test -z \"${test_tick+set}\"\n+\tthen\n+\t\ttest_tick=1112911993\n+\telse\n+\t\ttest_tick=$(($test_tick + 60))\n+\tfi\n+\tGIT_COMMITTER_DATE=\"$test_tick -0700\"\n+\tGIT_AUTHOR_DATE=\"$test_tick -0700\"\n+\texport GIT_COMMITTER_DATE GIT_AUTHOR_DATE\n+}\n+\n+# Call test_commit with the arguments \"<message> [<file> [<contents>]]\"\n+#\n+# This will commit a file with the given contents and the given commit\n+# message.  It will also add a tag with <message> as name.\n+#\n+# Both <file> and <contents> default to <message>.\n+\n+test_commit () {\n+\tfile=${2:-\"$1.t\"}\n+\techo \"${3-$1}\" > \"$file\" &&\n+\tgit add \"$file\" &&\n+\ttest_tick &&\n+\tgit commit -m \"$1\" &&\n+\tgit tag \"$1\"\n+}\n+\n+# Call test_merge with the arguments \"<message> <commit>\", where <commit>\n+# can be a tag pointing to the commit-to-merge.\n+\n+test_merge () {\n+\ttest_tick &&\n+\tgit merge -m \"$1\" \"$2\" &&\n+\tgit tag \"$1\"\n+}\n+\n+# This function helps systems where core.filemode=false is set.\n+# Use it instead of plain 'chmod +x' to set or unset the executable bit\n+# of a file in the working directory and add it to the index.\n+\n+test_chmod () {\n+\tchmod \"$@\" &&\n+\tgit update-index --add \"--chmod=$@\"\n+}\n+\n+# Unset a configuration variable, but don't fail if it doesn't exist.\n+test_unconfig () {\n+\tgit config --unset-all \"$@\"\n+\tconfig_status=$?\n+\tcase \"$config_status\" in\n+\t5) # ok, nothing to unset\n+\t\tconfig_status=0\n+\t\t;;\n+\tesac\n+\treturn $config_status\n+}\n+\n+# Set git config, automatically unsetting it after the test is over.\n+test_config () {\n+\ttest_when_finished \"test_unconfig '$1'\" &&\n+\tgit config \"$@\"\n+}\n+\n+test_config_global () {\n+\ttest_when_finished \"test_unconfig --global '$1'\" &&\n+\tgit config --global \"$@\"\n+}\n+\n+# Use test_set_prereq to tell that a particular prerequisite is available.\n+# The prerequisite can later be checked for in two ways:\n+#\n+# - Explicitly using test_have_prereq.\n+#\n+# - Implicitly by specifying the prerequisite tag in the calls to\n+#   test_expect_{success,failure,code}.\n+#\n+# The single parameter is the prerequisite tag (a simple word, in all\n+# capital letters by convention).\n+\n+test_set_prereq () {\n+\tsatisfied=\"$satisfied$1 \"\n+}\n+satisfied=\" \"\n+\n+test_have_prereq () {\n+\t# prerequisites can be concatenated with ','\n+\tsave_IFS=$IFS\n+\tIFS=,\n+\tset -- $*\n+\tIFS=$save_IFS\n+\n+\ttotal_prereq=0\n+\tok_prereq=0\n+\tmissing_prereq=\n+\n+\tfor prerequisite\n+\tdo\n+\t\ttotal_prereq=$(($total_prereq + 1))\n+\t\tcase $satisfied in\n+\t\t*\" $prerequisite \"*)\n+\t\t\tok_prereq=$(($ok_prereq + 1))\n+\t\t\t;;\n+\t\t*)\n+\t\t\t# Keep a list of missing prerequisites\n+\t\t\tif test -z \"$missing_prereq\"\n+\t\t\tthen\n+\t\t\t\tmissing_prereq=$prerequisite\n+\t\t\telse\n+\t\t\t\tmissing_prereq=\"$prerequisite,$missing_prereq\"\n+\t\t\tfi\n+\t\tesac\n+\tdone\n+\n+\ttest $total_prereq = $ok_prereq\n+}\n+\n+test_declared_prereq () {\n+\tcase \",$test_prereq,\" in\n+\t*,$1,*)\n+\t\treturn 0\n+\t\t;;\n+\tesac\n+\treturn 1\n+}\n+\n+test_expect_failure () {\n+\ttest \"$#\" = 3 && { test_prereq=$1; shift; } || test_prereq=\n+\ttest \"$#\" = 2 ||\n+\terror \"bug in the test script: not 2 or 3 parameters to test-expect-failure\"\n+\texport test_prereq\n+\tif ! test_skip \"$@\"\n+\tthen\n+\t\tsay >&3 \"checking known breakage: $2\"\n+\t\tif test_run_ \"$2\" expecting_failure\n+\t\tthen\n+\t\t\ttest_known_broken_ok_ \"$1\"\n+\t\telse\n+\t\t\ttest_known_broken_failure_ \"$1\"\n+\t\tfi\n+\tfi\n+\techo >&3 \"\"\n+}\n+\n+test_expect_success () {\n+\ttest \"$#\" = 3 && { test_prereq=$1; shift; } || test_prereq=\n+\ttest \"$#\" = 2 ||\n+\terror \"bug in the test script: not 2 or 3 parameters to test-expect-success\"\n+\texport test_prereq\n+\tif ! test_skip \"$@\"\n+\tthen\n+\t\tsay >&3 \"expecting success: $2\"\n+\t\tif test_run_ \"$2\"\n+\t\tthen\n+\t\t\ttest_ok_ \"$1\"\n+\t\telse\n+\t\t\ttest_failure_ \"$@\"\n+\t\tfi\n+\tfi\n+\techo >&3 \"\"\n+}\n+\n+# test_external runs external test scripts that provide continuous\n+# test output about their progress, and succeeds/fails on\n+# zero/non-zero exit code.  It outputs the test output on stdout even\n+# in non-verbose mode, and announces the external script with \"# run\n+# <n>: ...\" before running it.  When providing relative paths, keep in\n+# mind that all scripts run in \"trash directory\".\n+# Usage: test_external description command arguments...\n+# Example: test_external 'Perl API' perl ../path/to/test.pl\n+test_external () {\n+\ttest \"$#\" = 4 && { test_prereq=$1; shift; } || test_prereq=\n+\ttest \"$#\" = 3 ||\n+\terror >&5 \"bug in the test script: not 3 or 4 parameters to test_external\"\n+\tdescr=\"$1\"\n+\tshift\n+\texport test_prereq\n+\tif ! test_skip \"$descr\" \"$@\"\n+\tthen\n+\t\t# Announce the script to reduce confusion about the\n+\t\t# test output that follows.\n+\t\tsay_color \"\" \"# run $test_count: $descr ($*)\"\n+\t\t# Export TEST_DIRECTORY, TRASH_DIRECTORY and GIT_TEST_LONG\n+\t\t# to be able to use them in script\n+\t\texport TEST_DIRECTORY TRASH_DIRECTORY GIT_TEST_LONG\n+\t\t# Run command; redirect its stderr to &4 as in\n+\t\t# test_run_, but keep its stdout on our stdout even in\n+\t\t# non-verbose mode.\n+\t\t\"$@\" 2>&4\n+\t\tif [ \"$?\" = 0 ]\n+\t\tthen\n+\t\t\tif test $test_external_has_tap -eq 0; then\n+\t\t\t\ttest_ok_ \"$descr\"\n+\t\t\telse\n+\t\t\t\tsay_color \"\" \"# test_external test $descr was ok\"\n+\t\t\t\ttest_success=$(($test_success + 1))\n+\t\t\tfi\n+\t\telse\n+\t\t\tif test $test_external_has_tap -eq 0; then\n+\t\t\t\ttest_failure_ \"$descr\" \"$@\"\n+\t\t\telse\n+\t\t\t\tsay_color error \"# test_external test $descr failed: $@\"\n+\t\t\t\ttest_failure=$(($test_failure + 1))\n+\t\t\tfi\n+\t\tfi\n+\tfi\n+}\n+\n+# Like test_external, but in addition tests that the command generated\n+# no output on stderr.\n+test_external_without_stderr () {\n+\t# The temporary file has no (and must have no) security\n+\t# implications.\n+\ttmp=${TMPDIR:-/tmp}\n+\tstderr=\"$tmp/git-external-stderr.$$.tmp\"\n+\ttest_external \"$@\" 4> \"$stderr\"\n+\t[ -f \"$stderr\" ] || error \"Internal error: $stderr disappeared.\"\n+\tdescr=\"no stderr: $1\"\n+\tshift\n+\tsay >&3 \"# expecting no stderr from previous command\"\n+\tif [ ! -s \"$stderr\" ]; then\n+\t\trm \"$stderr\"\n+\n+\t\tif test $test_external_has_tap -eq 0; then\n+\t\t\ttest_ok_ \"$descr\"\n+\t\telse\n+\t\t\tsay_color \"\" \"# test_external_without_stderr test $descr was ok\"\n+\t\t\ttest_success=$(($test_success + 1))\n+\t\tfi\n+\telse\n+\t\tif [ \"$verbose\" = t ]; then\n+\t\t\toutput=`echo; echo \"# Stderr is:\"; cat \"$stderr\"`\n+\t\telse\n+\t\t\toutput=\n+\t\tfi\n+\t\t# rm first in case test_failure exits.\n+\t\trm \"$stderr\"\n+\t\tif test $test_external_has_tap -eq 0; then\n+\t\t\ttest_failure_ \"$descr\" \"$@\" \"$output\"\n+\t\telse\n+\t\t\tsay_color error \"# test_external_without_stderr test $descr failed: $@: $output\"\n+\t\t\ttest_failure=$(($test_failure + 1))\n+\t\tfi\n+\tfi\n+}\n+\n+# debugging-friendly alternatives to \"test [-f|-d|-e]\"\n+# The commands test the existence or non-existence of $1. $2 can be\n+# given to provide a more precise diagnosis.\n+test_path_is_file () {\n+\tif ! [ -f \"$1\" ]\n+\tthen\n+\t\techo \"File $1 doesn't exist. $*\"\n+\t\tfalse\n+\tfi\n+}\n+\n+test_path_is_dir () {\n+\tif ! [ -d \"$1\" ]\n+\tthen\n+\t\techo \"Directory $1 doesn't exist. $*\"\n+\t\tfalse\n+\tfi\n+}\n+\n+test_path_is_missing () {\n+\tif [ -e \"$1\" ]\n+\tthen\n+\t\techo \"Path exists:\"\n+\t\tls -ld \"$1\"\n+\t\tif [ $# -ge 1 ]; then\n+\t\t\techo \"$*\"\n+\t\tfi\n+\t\tfalse\n+\tfi\n+}\n+\n+# test_line_count checks that a file has the number of lines it\n+# ought to. For example:\n+#\n+#\ttest_expect_success 'produce exactly one line of output' '\n+#\t\tdo something >output &&\n+#\t\ttest_line_count = 1 output\n+#\t'\n+#\n+# is like \"test $(wc -l <output) = 1\" except that it passes the\n+# output through when the number of lines is wrong.\n+\n+test_line_count () {\n+\tif test $# != 3\n+\tthen\n+\t\terror \"bug in the test script: not 3 parameters to test_line_count\"\n+\telif ! test $(wc -l <\"$3\") \"$1\" \"$2\"\n+\tthen\n+\t\techo \"test_line_count: line count for $3 !$1 $2\"\n+\t\tcat \"$3\"\n+\t\treturn 1\n+\tfi\n+}\n+\n+# This is not among top-level (test_expect_success | test_expect_failure)\n+# but is a prefix that can be used in the test script, like:\n+#\n+#\ttest_expect_success 'complain and die' '\n+#           do something &&\n+#           do something else &&\n+#\t    test_must_fail git checkout ../outerspace\n+#\t'\n+#\n+# Writing this as \"! git checkout ../outerspace\" is wrong, because\n+# the failure could be due to a segv.  We want a controlled failure.\n+\n+test_must_fail () {\n+\t\"$@\"\n+\texit_code=$?\n+\tif test $exit_code = 0; then\n+\t\techo >&2 \"test_must_fail: command succeeded: $*\"\n+\t\treturn 1\n+\telif test $exit_code -gt 129 -a $exit_code -le 192; then\n+\t\techo >&2 \"test_must_fail: died by signal: $*\"\n+\t\treturn 1\n+\telif test $exit_code = 127; then\n+\t\techo >&2 \"test_must_fail: command not found: $*\"\n+\t\treturn 1\n+\tfi\n+\treturn 0\n+}\n+\n+# Similar to test_must_fail, but tolerates success, too.  This is\n+# meant to be used in contexts like:\n+#\n+#\ttest_expect_success 'some command works without configuration' '\n+#\t\ttest_might_fail git config --unset all.configuration &&\n+#\t\tdo something\n+#\t'\n+#\n+# Writing \"git config --unset all.configuration || :\" would be wrong,\n+# because we want to notice if it fails due to segv.\n+\n+test_might_fail () {\n+\t\"$@\"\n+\texit_code=$?\n+\tif test $exit_code -gt 129 -a $exit_code -le 192; then\n+\t\techo >&2 \"test_might_fail: died by signal: $*\"\n+\t\treturn 1\n+\telif test $exit_code = 127; then\n+\t\techo >&2 \"test_might_fail: command not found: $*\"\n+\t\treturn 1\n+\tfi\n+\treturn 0\n+}\n+\n+# Similar to test_must_fail and test_might_fail, but check that a\n+# given command exited with a given exit code. Meant to be used as:\n+#\n+#\ttest_expect_success 'Merge with d/f conflicts' '\n+#\t\ttest_expect_code 1 git merge \"merge msg\" B master\n+#\t'\n+\n+test_expect_code () {\n+\twant_code=$1\n+\tshift\n+\t\"$@\"\n+\texit_code=$?\n+\tif test $exit_code = $want_code\n+\tthen\n+\t\treturn 0\n+\tfi\n+\n+\techo >&2 \"test_expect_code: command exited with $exit_code, we wanted $want_code $*\"\n+\treturn 1\n+}\n+\n+# test_cmp is a helper function to compare actual and expected output.\n+# You can use it like:\n+#\n+#\ttest_expect_success 'foo works' '\n+#\t\techo expected >expected &&\n+#\t\tfoo >actual &&\n+#\t\ttest_cmp expected actual\n+#\t'\n+#\n+# This could be written as either \"cmp\" or \"diff -u\", but:\n+# - cmp's output is not nearly as easy to read as diff -u\n+# - not all diff versions understand \"-u\"\n+\n+test_cmp() {\n+\t$GIT_TEST_CMP \"$@\"\n+}\n+\n+# This function can be used to schedule some commands to be run\n+# unconditionally at the end of the test to restore sanity:\n+#\n+#\ttest_expect_success 'test core.capslock' '\n+#\t\tgit config core.capslock true &&\n+#\t\ttest_when_finished \"git config --unset core.capslock\" &&\n+#\t\thello world\n+#\t'\n+#\n+# That would be roughly equivalent to\n+#\n+#\ttest_expect_success 'test core.capslock' '\n+#\t\tgit config core.capslock true &&\n+#\t\thello world\n+#\t\tgit config --unset core.capslock\n+#\t'\n+#\n+# except that the greeting and config --unset must both succeed for\n+# the test to pass.\n+#\n+# Note that under --immediate mode, no clean-up is done to help diagnose\n+# what went wrong.\n+\n+test_when_finished () {\n+\ttest_cleanup=\"{ $*\n+\t\t} && (exit \\\"\\$eval_ret\\\"); eval_ret=\\$?; $test_cleanup\"\n+}\n+\n+# Most tests can use the created repository, but some may need to create more.\n+# Usage: test_create_repo <directory>\n+test_create_repo () {\n+\ttest \"$#\" = 1 ||\n+\terror \"bug in the test script: not 1 parameter to test-create-repo\"\n+\trepo=\"$1\"\n+\tmkdir -p \"$repo\"\n+\t(\n+\t\tcd \"$repo\" || error \"Cannot setup test environment\"\n+\t\t\"$GIT_EXEC_PATH/git-init\" \"--template=$GIT_BUILD_DIR/templates/blt/\" >&3 2>&4 ||\n+\t\terror \"cannot run git init -- have you built things yet?\"\n+\t\tmv .git/hooks .git/hooks-disabled\n+\t) || exit\n+}\n+\ndiff --git a/t/test-lib.sh b/t/test-lib.sh\nindex 160479b..368e48c 100644\n--- a/t/test-lib.sh\n+++ b/t/test-lib.sh\n@@ -96,6 +96,8 @@ _z40=0000000000000000000000000000000000000000\n LF='\n '\n \n+export _x05 _x40 _z40 LF\n+\n # Each test should start with something like this, after copyright notices:\n #\n # test_description='Description of this test...\n@@ -220,226 +222,9 @@ die () {\n GIT_EXIT_OK=\n trap 'die' EXIT\n \n-# The semantics of the editor variables are that of invoking\n-# sh -c \"$EDITOR \\\"$@\\\"\" files ...\n-#\n-# If our trash directory contains shell metacharacters, they will be\n-# interpreted if we just set $EDITOR directly, so do a little dance with\n-# environment variables to work around this.\n-#\n-# In particular, quoting isn't enough, as the path may contain the same quote\n-# that we're using.\n-test_set_editor () {\n-\tFAKE_EDITOR=\"$1\"\n-\texport FAKE_EDITOR\n-\tEDITOR='\"$FAKE_EDITOR\"'\n-\texport EDITOR\n-}\n-\n-test_decode_color () {\n-\tawk '\n-\t\tfunction name(n) {\n-\t\t\tif (n == 0) return \"RESET\";\n-\t\t\tif (n == 1) return \"BOLD\";\n-\t\t\tif (n == 30) return \"BLACK\";\n-\t\t\tif (n == 31) return \"RED\";\n-\t\t\tif (n == 32) return \"GREEN\";\n-\t\t\tif (n == 33) return \"YELLOW\";\n-\t\t\tif (n == 34) return \"BLUE\";\n-\t\t\tif (n == 35) return \"MAGENTA\";\n-\t\t\tif (n == 36) return \"CYAN\";\n-\t\t\tif (n == 37) return \"WHITE\";\n-\t\t\tif (n == 40) return \"BLACK\";\n-\t\t\tif (n == 41) return \"BRED\";\n-\t\t\tif (n == 42) return \"BGREEN\";\n-\t\t\tif (n == 43) return \"BYELLOW\";\n-\t\t\tif (n == 44) return \"BBLUE\";\n-\t\t\tif (n == 45) return \"BMAGENTA\";\n-\t\t\tif (n == 46) return \"BCYAN\";\n-\t\t\tif (n == 47) return \"BWHITE\";\n-\t\t}\n-\t\t{\n-\t\t\twhile (match($0, /\\033\\[[0-9;]*m/) != 0) {\n-\t\t\t\tprintf \"%s<\", substr($0, 1, RSTART-1);\n-\t\t\t\tcodes = substr($0, RSTART+2, RLENGTH-3);\n-\t\t\t\tif (length(codes) == 0)\n-\t\t\t\t\tprintf \"%s\", name(0)\n-\t\t\t\telse {\n-\t\t\t\t\tn = split(codes, ary, \";\");\n-\t\t\t\t\tsep = \"\";\n-\t\t\t\t\tfor (i = 1; i <= n; i++) {\n-\t\t\t\t\t\tprintf \"%s%s\", sep, name(ary[i]);\n-\t\t\t\t\t\tsep = \";\"\n-\t\t\t\t\t}\n-\t\t\t\t}\n-\t\t\t\tprintf \">\";\n-\t\t\t\t$0 = substr($0, RSTART + RLENGTH, length($0) - RSTART - RLENGTH + 1);\n-\t\t\t}\n-\t\t\tprint\n-\t\t}\n-\t'\n-}\n-\n-nul_to_q () {\n-\tperl -pe 'y/\\000/Q/'\n-}\n-\n-q_to_nul () {\n-\tperl -pe 'y/Q/\\000/'\n-}\n-\n-q_to_cr () {\n-\ttr Q '\\015'\n-}\n-\n-q_to_tab () {\n-\ttr Q '\\011'\n-}\n-\n-append_cr () {\n-\tsed -e 's/$/Q/' | tr Q '\\015'\n-}\n-\n-remove_cr () {\n-\ttr '\\015' Q | sed -e 's/Q$//'\n-}\n-\n-# In some bourne shell implementations, the \"unset\" builtin returns\n-# nonzero status when a variable to be unset was not set in the first\n-# place.\n-#\n-# Use sane_unset when that should not be considered an error.\n-\n-sane_unset () {\n-\tunset \"$@\"\n-\treturn 0\n-}\n-\n-test_tick () {\n-\tif test -z \"${test_tick+set}\"\n-\tthen\n-\t\ttest_tick=1112911993\n-\telse\n-\t\ttest_tick=$(($test_tick + 60))\n-\tfi\n-\tGIT_COMMITTER_DATE=\"$test_tick -0700\"\n-\tGIT_AUTHOR_DATE=\"$test_tick -0700\"\n-\texport GIT_COMMITTER_DATE GIT_AUTHOR_DATE\n-}\n-\n-# Call test_commit with the arguments \"<message> [<file> [<contents>]]\"\n-#\n-# This will commit a file with the given contents and the given commit\n-# message.  It will also add a tag with <message> as name.\n-#\n-# Both <file> and <contents> default to <message>.\n-\n-test_commit () {\n-\tfile=${2:-\"$1.t\"}\n-\techo \"${3-$1}\" > \"$file\" &&\n-\tgit add \"$file\" &&\n-\ttest_tick &&\n-\tgit commit -m \"$1\" &&\n-\tgit tag \"$1\"\n-}\n-\n-# Call test_merge with the arguments \"<message> <commit>\", where <commit>\n-# can be a tag pointing to the commit-to-merge.\n-\n-test_merge () {\n-\ttest_tick &&\n-\tgit merge -m \"$1\" \"$2\" &&\n-\tgit tag \"$1\"\n-}\n-\n-# This function helps systems where core.filemode=false is set.\n-# Use it instead of plain 'chmod +x' to set or unset the executable bit\n-# of a file in the working directory and add it to the index.\n-\n-test_chmod () {\n-\tchmod \"$@\" &&\n-\tgit update-index --add \"--chmod=$@\"\n-}\n-\n-# Unset a configuration variable, but don't fail if it doesn't exist.\n-test_unconfig () {\n-\tgit config --unset-all \"$@\"\n-\tconfig_status=$?\n-\tcase \"$config_status\" in\n-\t5) # ok, nothing to unset\n-\t\tconfig_status=0\n-\t\t;;\n-\tesac\n-\treturn $config_status\n-}\n-\n-# Set git config, automatically unsetting it after the test is over.\n-test_config () {\n-\ttest_when_finished \"test_unconfig '$1'\" &&\n-\tgit config \"$@\"\n-}\n-\n-test_config_global () {\n-\ttest_when_finished \"test_unconfig --global '$1'\" &&\n-\tgit config --global \"$@\"\n-}\n-\n-# Use test_set_prereq to tell that a particular prerequisite is available.\n-# The prerequisite can later be checked for in two ways:\n-#\n-# - Explicitly using test_have_prereq.\n-#\n-# - Implicitly by specifying the prerequisite tag in the calls to\n-#   test_expect_{success,failure,code}.\n-#\n-# The single parameter is the prerequisite tag (a simple word, in all\n-# capital letters by convention).\n-\n-test_set_prereq () {\n-\tsatisfied=\"$satisfied$1 \"\n-}\n-satisfied=\" \"\n-\n-test_have_prereq () {\n-\t# prerequisites can be concatenated with ','\n-\tsave_IFS=$IFS\n-\tIFS=,\n-\tset -- $*\n-\tIFS=$save_IFS\n-\n-\ttotal_prereq=0\n-\tok_prereq=0\n-\tmissing_prereq=\n-\n-\tfor prerequisite\n-\tdo\n-\t\ttotal_prereq=$(($total_prereq + 1))\n-\t\tcase $satisfied in\n-\t\t*\" $prerequisite \"*)\n-\t\t\tok_prereq=$(($ok_prereq + 1))\n-\t\t\t;;\n-\t\t*)\n-\t\t\t# Keep a list of missing prerequisites\n-\t\t\tif test -z \"$missing_prereq\"\n-\t\t\tthen\n-\t\t\t\tmissing_prereq=$prerequisite\n-\t\t\telse\n-\t\t\t\tmissing_prereq=\"$prerequisite,$missing_prereq\"\n-\t\t\tfi\n-\t\tesac\n-\tdone\n-\n-\ttest $total_prereq = $ok_prereq\n-}\n-\n-test_declared_prereq () {\n-\tcase \",$test_prereq,\" in\n-\t*,$1,*)\n-\t\treturn 0\n-\t\t;;\n-\tesac\n-\treturn 1\n-}\n+# The user-facing functions are loaded from a separate file so that\n+# test_perf subshells can have them too\n+. \"${TEST_DIRECTORY:-.}\"/test-lib-functions.sh\n \n # You are not expected to call test_ok_ and test_failure_ directly, use\n # the text_expect_* functions instead.\n@@ -527,311 +312,6 @@ test_skip () {\n \tesac\n }\n \n-test_expect_failure () {\n-\ttest \"$#\" = 3 && { test_prereq=$1; shift; } || test_prereq=\n-\ttest \"$#\" = 2 ||\n-\terror \"bug in the test script: not 2 or 3 parameters to test-expect-failure\"\n-\texport test_prereq\n-\tif ! test_skip \"$@\"\n-\tthen\n-\t\tsay >&3 \"checking known breakage: $2\"\n-\t\tif test_run_ \"$2\" expecting_failure\n-\t\tthen\n-\t\t\ttest_known_broken_ok_ \"$1\"\n-\t\telse\n-\t\t\ttest_known_broken_failure_ \"$1\"\n-\t\tfi\n-\tfi\n-\techo >&3 \"\"\n-}\n-\n-test_expect_success () {\n-\ttest \"$#\" = 3 && { test_prereq=$1; shift; } || test_prereq=\n-\ttest \"$#\" = 2 ||\n-\terror \"bug in the test script: not 2 or 3 parameters to test-expect-success\"\n-\texport test_prereq\n-\tif ! test_skip \"$@\"\n-\tthen\n-\t\tsay >&3 \"expecting success: $2\"\n-\t\tif test_run_ \"$2\"\n-\t\tthen\n-\t\t\ttest_ok_ \"$1\"\n-\t\telse\n-\t\t\ttest_failure_ \"$@\"\n-\t\tfi\n-\tfi\n-\techo >&3 \"\"\n-}\n-\n-# test_external runs external test scripts that provide continuous\n-# test output about their progress, and succeeds/fails on\n-# zero/non-zero exit code.  It outputs the test output on stdout even\n-# in non-verbose mode, and announces the external script with \"# run\n-# <n>: ...\" before running it.  When providing relative paths, keep in\n-# mind that all scripts run in \"trash directory\".\n-# Usage: test_external description command arguments...\n-# Example: test_external 'Perl API' perl ../path/to/test.pl\n-test_external () {\n-\ttest \"$#\" = 4 && { test_prereq=$1; shift; } || test_prereq=\n-\ttest \"$#\" = 3 ||\n-\terror >&5 \"bug in the test script: not 3 or 4 parameters to test_external\"\n-\tdescr=\"$1\"\n-\tshift\n-\texport test_prereq\n-\tif ! test_skip \"$descr\" \"$@\"\n-\tthen\n-\t\t# Announce the script to reduce confusion about the\n-\t\t# test output that follows.\n-\t\tsay_color \"\" \"# run $test_count: $descr ($*)\"\n-\t\t# Export TEST_DIRECTORY, TRASH_DIRECTORY and GIT_TEST_LONG\n-\t\t# to be able to use them in script\n-\t\texport TEST_DIRECTORY TRASH_DIRECTORY GIT_TEST_LONG\n-\t\t# Run command; redirect its stderr to &4 as in\n-\t\t# test_run_, but keep its stdout on our stdout even in\n-\t\t# non-verbose mode.\n-\t\t\"$@\" 2>&4\n-\t\tif [ \"$?\" = 0 ]\n-\t\tthen\n-\t\t\tif test $test_external_has_tap -eq 0; then\n-\t\t\t\ttest_ok_ \"$descr\"\n-\t\t\telse\n-\t\t\t\tsay_color \"\" \"# test_external test $descr was ok\"\n-\t\t\t\ttest_success=$(($test_success + 1))\n-\t\t\tfi\n-\t\telse\n-\t\t\tif test $test_external_has_tap -eq 0; then\n-\t\t\t\ttest_failure_ \"$descr\" \"$@\"\n-\t\t\telse\n-\t\t\t\tsay_color error \"# test_external test $descr failed: $@\"\n-\t\t\t\ttest_failure=$(($test_failure + 1))\n-\t\t\tfi\n-\t\tfi\n-\tfi\n-}\n-\n-# Like test_external, but in addition tests that the command generated\n-# no output on stderr.\n-test_external_without_stderr () {\n-\t# The temporary file has no (and must have no) security\n-\t# implications.\n-\ttmp=${TMPDIR:-/tmp}\n-\tstderr=\"$tmp/git-external-stderr.$$.tmp\"\n-\ttest_external \"$@\" 4> \"$stderr\"\n-\t[ -f \"$stderr\" ] || error \"Internal error: $stderr disappeared.\"\n-\tdescr=\"no stderr: $1\"\n-\tshift\n-\tsay >&3 \"# expecting no stderr from previous command\"\n-\tif [ ! -s \"$stderr\" ]; then\n-\t\trm \"$stderr\"\n-\n-\t\tif test $test_external_has_tap -eq 0; then\n-\t\t\ttest_ok_ \"$descr\"\n-\t\telse\n-\t\t\tsay_color \"\" \"# test_external_without_stderr test $descr was ok\"\n-\t\t\ttest_success=$(($test_success + 1))\n-\t\tfi\n-\telse\n-\t\tif [ \"$verbose\" = t ]; then\n-\t\t\toutput=`echo; echo \"# Stderr is:\"; cat \"$stderr\"`\n-\t\telse\n-\t\t\toutput=\n-\t\tfi\n-\t\t# rm first in case test_failure exits.\n-\t\trm \"$stderr\"\n-\t\tif test $test_external_has_tap -eq 0; then\n-\t\t\ttest_failure_ \"$descr\" \"$@\" \"$output\"\n-\t\telse\n-\t\t\tsay_color error \"# test_external_without_stderr test $descr failed: $@: $output\"\n-\t\t\ttest_failure=$(($test_failure + 1))\n-\t\tfi\n-\tfi\n-}\n-\n-# debugging-friendly alternatives to \"test [-f|-d|-e]\"\n-# The commands test the existence or non-existence of $1. $2 can be\n-# given to provide a more precise diagnosis.\n-test_path_is_file () {\n-\tif ! [ -f \"$1\" ]\n-\tthen\n-\t\techo \"File $1 doesn't exist. $*\"\n-\t\tfalse\n-\tfi\n-}\n-\n-test_path_is_dir () {\n-\tif ! [ -d \"$1\" ]\n-\tthen\n-\t\techo \"Directory $1 doesn't exist. $*\"\n-\t\tfalse\n-\tfi\n-}\n-\n-test_path_is_missing () {\n-\tif [ -e \"$1\" ]\n-\tthen\n-\t\techo \"Path exists:\"\n-\t\tls -ld \"$1\"\n-\t\tif [ $# -ge 1 ]; then\n-\t\t\techo \"$*\"\n-\t\tfi\n-\t\tfalse\n-\tfi\n-}\n-\n-# test_line_count checks that a file has the number of lines it\n-# ought to. For example:\n-#\n-#\ttest_expect_success 'produce exactly one line of output' '\n-#\t\tdo something >output &&\n-#\t\ttest_line_count = 1 output\n-#\t'\n-#\n-# is like \"test $(wc -l <output) = 1\" except that it passes the\n-# output through when the number of lines is wrong.\n-\n-test_line_count () {\n-\tif test $# != 3\n-\tthen\n-\t\terror \"bug in the test script: not 3 parameters to test_line_count\"\n-\telif ! test $(wc -l <\"$3\") \"$1\" \"$2\"\n-\tthen\n-\t\techo \"test_line_count: line count for $3 !$1 $2\"\n-\t\tcat \"$3\"\n-\t\treturn 1\n-\tfi\n-}\n-\n-# This is not among top-level (test_expect_success | test_expect_failure)\n-# but is a prefix that can be used in the test script, like:\n-#\n-#\ttest_expect_success 'complain and die' '\n-#           do something &&\n-#           do something else &&\n-#\t    test_must_fail git checkout ../outerspace\n-#\t'\n-#\n-# Writing this as \"! git checkout ../outerspace\" is wrong, because\n-# the failure could be due to a segv.  We want a controlled failure.\n-\n-test_must_fail () {\n-\t\"$@\"\n-\texit_code=$?\n-\tif test $exit_code = 0; then\n-\t\techo >&2 \"test_must_fail: command succeeded: $*\"\n-\t\treturn 1\n-\telif test $exit_code -gt 129 -a $exit_code -le 192; then\n-\t\techo >&2 \"test_must_fail: died by signal: $*\"\n-\t\treturn 1\n-\telif test $exit_code = 127; then\n-\t\techo >&2 \"test_must_fail: command not found: $*\"\n-\t\treturn 1\n-\tfi\n-\treturn 0\n-}\n-\n-# Similar to test_must_fail, but tolerates success, too.  This is\n-# meant to be used in contexts like:\n-#\n-#\ttest_expect_success 'some command works without configuration' '\n-#\t\ttest_might_fail git config --unset all.configuration &&\n-#\t\tdo something\n-#\t'\n-#\n-# Writing \"git config --unset all.configuration || :\" would be wrong,\n-# because we want to notice if it fails due to segv.\n-\n-test_might_fail () {\n-\t\"$@\"\n-\texit_code=$?\n-\tif test $exit_code -gt 129 -a $exit_code -le 192; then\n-\t\techo >&2 \"test_might_fail: died by signal: $*\"\n-\t\treturn 1\n-\telif test $exit_code = 127; then\n-\t\techo >&2 \"test_might_fail: command not found: $*\"\n-\t\treturn 1\n-\tfi\n-\treturn 0\n-}\n-\n-# Similar to test_must_fail and test_might_fail, but check that a\n-# given command exited with a given exit code. Meant to be used as:\n-#\n-#\ttest_expect_success 'Merge with d/f conflicts' '\n-#\t\ttest_expect_code 1 git merge \"merge msg\" B master\n-#\t'\n-\n-test_expect_code () {\n-\twant_code=$1\n-\tshift\n-\t\"$@\"\n-\texit_code=$?\n-\tif test $exit_code = $want_code\n-\tthen\n-\t\treturn 0\n-\tfi\n-\n-\techo >&2 \"test_expect_code: command exited with $exit_code, we wanted $want_code $*\"\n-\treturn 1\n-}\n-\n-# test_cmp is a helper function to compare actual and expected output.\n-# You can use it like:\n-#\n-#\ttest_expect_success 'foo works' '\n-#\t\techo expected >expected &&\n-#\t\tfoo >actual &&\n-#\t\ttest_cmp expected actual\n-#\t'\n-#\n-# This could be written as either \"cmp\" or \"diff -u\", but:\n-# - cmp's output is not nearly as easy to read as diff -u\n-# - not all diff versions understand \"-u\"\n-\n-test_cmp() {\n-\t$GIT_TEST_CMP \"$@\"\n-}\n-\n-# This function can be used to schedule some commands to be run\n-# unconditionally at the end of the test to restore sanity:\n-#\n-#\ttest_expect_success 'test core.capslock' '\n-#\t\tgit config core.capslock true &&\n-#\t\ttest_when_finished \"git config --unset core.capslock\" &&\n-#\t\thello world\n-#\t'\n-#\n-# That would be roughly equivalent to\n-#\n-#\ttest_expect_success 'test core.capslock' '\n-#\t\tgit config core.capslock true &&\n-#\t\thello world\n-#\t\tgit config --unset core.capslock\n-#\t'\n-#\n-# except that the greeting and config --unset must both succeed for\n-# the test to pass.\n-#\n-# Note that under --immediate mode, no clean-up is done to help diagnose\n-# what went wrong.\n-\n-test_when_finished () {\n-\ttest_cleanup=\"{ $*\n-\t\t} && (exit \\\"\\$eval_ret\\\"); eval_ret=\\$?; $test_cleanup\"\n-}\n-\n-# Most tests can use the created repository, but some may need to create more.\n-# Usage: test_create_repo <directory>\n-test_create_repo () {\n-\ttest \"$#\" = 1 ||\n-\terror \"bug in the test script: not 1 parameter to test-create-repo\"\n-\trepo=\"$1\"\n-\tmkdir -p \"$repo\"\n-\t(\n-\t\tcd \"$repo\" || error \"Cannot setup test environment\"\n-\t\t\"$GIT_EXEC_PATH/git-init\" \"--template=$GIT_BUILD_DIR/templates/blt/\" >&3 2>&4 ||\n-\t\terror \"cannot run git init -- have you built things yet?\"\n-\t\tmv .git/hooks .git/hooks-disabled\n-\t) || exit\n }\n \n test_done () {\n-- \n1.7.8.304.ge42e4\n"},{"id":"181146","messageId":"94f64a03398829bb9a11c18577efb39d9b153eca.1323876121.git.trast@student.ethz.ch","threadId":"29165","inReplyTo":"cover.1323876121.git.trast@student.ethz.ch","subject":"[RFC PATCH 2/4] test-lib: allow testing another git build tree","fromName":"Thomas Rast","fromEmail":"trast@student.ethz.ch","sentAt":"2011-12-14T15:23:30Z","receivedAt":"2011-12-14T15:23:30Z","isPatch":true,"sender":{"key":"tr@thomasrast.ch","avatar":"https://avatars.githubusercontent.com/u/153510?v=4"},"body":"The perf-lib work wants this feature, so we may as well do it for\ntest-lib in general.  You can now say\n\n  GIT_BUILD_DIR=/another/git/tree\n  make test  # or any other test\n\nand it will run the tests of the current tree against the binaries of\nthe other tree.\n\nSigned-off-by: Thomas Rast <trast@student.ethz.ch>\n---\n Makefile      |    7 ++++++-\n t/test-lib.sh |   19 ++++++++++++++++---\n 2 files changed, 22 insertions(+), 4 deletions(-)\n\ndiff --git a/Makefile b/Makefile\nindex 2127c1b..52506bf 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -2215,6 +2215,12 @@ GIT-BUILD-OPTIONS: FORCE\n \t@echo USE_LIBPCRE=\\''$(subst ','\\'',$(subst ','\\'',$(USE_LIBPCRE)))'\\' >>$@\n \t@echo NO_PERL=\\''$(subst ','\\'',$(subst ','\\'',$(NO_PERL)))'\\' >>$@\n \t@echo NO_PYTHON=\\''$(subst ','\\'',$(subst ','\\'',$(NO_PYTHON)))'\\' >>$@\n+\t@echo NO_UNIX_SOCKETS=\\''$(subst ','\\'',$(subst ','\\'',$(NO_UNIX_SOCKETS)))'\\' >>$@\n+\n+GIT-TEST-OPTIONS: FORCE\n+ifdef GIT_TEST_OPTS\n+\t@echo GIT_TEST_OPTS=\\''$(subst ','\\'',$(subst ','\\'',$(GIT_TEST_OPTS)))'\\' >$@\n+endif\n ifdef GIT_TEST_CMP\n \t@echo GIT_TEST_CMP=\\''$(subst ','\\'',$(subst ','\\'',$(GIT_TEST_CMP)))'\\' >>$@\n endif\n@@ -2222,7 +2228,6 @@ ifdef GIT_TEST_CMP_USE_COPIED_CONTEXT\n \t@echo GIT_TEST_CMP_USE_COPIED_CONTEXT=YesPlease >>$@\n endif\n \t@echo GETTEXT_POISON=\\''$(subst ','\\'',$(subst ','\\'',$(GETTEXT_POISON)))'\\' >>$@\n-\t@echo NO_UNIX_SOCKETS=\\''$(subst ','\\'',$(subst ','\\'',$(NO_UNIX_SOCKETS)))'\\' >>$@\n \n ### Detect Tck/Tk interpreter path changes\n ifndef NO_TCLTK\ndiff --git a/t/test-lib.sh b/t/test-lib.sh\nindex 368e48c..06f8c96 100644\n--- a/t/test-lib.sh\n+++ b/t/test-lib.sh\n@@ -54,6 +54,7 @@ unset $(perl -e '\n \t\t.*_TEST\n \t\tPROVE\n \t\tVALGRIND\n+\t\tBUILD_DIR\n \t));\n \tmy @vars = grep(/^GIT_/ && !/^GIT_($ok)/o, @env);\n \tprint join(\"\\n\", @vars);\n@@ -318,7 +319,7 @@ test_done () {\n \tGIT_EXIT_OK=t\n \n \tif test -z \"$HARNESS_ACTIVE\"; then\n-\t\ttest_results_dir=\"$TEST_DIRECTORY/test-results\"\n+\t\ttest_results_dir=\"$TEST_OUTPUT_DIRECTORY/test-results\"\n \t\tmkdir -p \"$test_results_dir\"\n \t\ttest_results_path=\"$test_results_dir/${0%.sh}-$$.counts\"\n \n@@ -379,7 +380,18 @@ then\n \t# itself.\n \tTEST_DIRECTORY=$(pwd)\n fi\n-GIT_BUILD_DIR=\"$TEST_DIRECTORY\"/..\n+if test -z \"$TEST_OUTPUT_DIRECTORY\"\n+then\n+\t# Similarly, override this to store the test-results subdir\n+\t# elsewhere\n+\tTEST_OUTPUT_DIRECTORY=$TEST_DIRECTORY\n+fi\n+if test -z \"$GIT_BUILD_DIR\"\n+then\n+\t# Similarly, override this to test a different git build tree\n+\t# than the one you are running the tests from\n+\tGIT_BUILD_DIR=\"$TEST_DIRECTORY\"/..\n+fi\n \n if test -n \"$valgrind\"\n then\n@@ -477,6 +489,7 @@ GIT_ATTR_NOSYSTEM=1\n export PATH GIT_EXEC_PATH GIT_TEMPLATE_DIR GIT_CONFIG_NOSYSTEM GIT_ATTR_NOSYSTEM\n \n . \"$GIT_BUILD_DIR\"/GIT-BUILD-OPTIONS\n+. \"$TEST_DIRECTORY\"/../GIT-TEST-OPTIONS\n \n if test -z \"$GIT_TEST_CMP\"\n then\n@@ -514,7 +527,7 @@ test=\"trash directory.$(basename \"$0\" .sh)\"\n test -n \"$root\" && test=\"$root/$test\"\n case \"$test\" in\n /*) TRASH_DIRECTORY=\"$test\" ;;\n- *) TRASH_DIRECTORY=\"$TEST_DIRECTORY/$test\" ;;\n+ *) TRASH_DIRECTORY=\"$TEST_OUTPUT_DIRECTORY/$test\" ;;\n esac\n test ! -z \"$debug\" || remove_trash=$TRASH_DIRECTORY\n rm -fr \"$test\" || {\n-- \n1.7.8.304.ge42e4\n"},{"id":"181148","messageId":"91630874c45015e74cf72df03b6011856fd0fe4f.1323876121.git.trast@student.ethz.ch","threadId":"29165","inReplyTo":"cover.1323876121.git.trast@student.ethz.ch","subject":"[RFC PATCH 3/4] Introduce a performance testing framework","fromName":"Thomas Rast","fromEmail":"trast@student.ethz.ch","sentAt":"2011-12-14T15:23:31Z","receivedAt":"2011-12-14T15:23:31Z","isPatch":true,"sender":{"key":"tr@thomasrast.ch","avatar":"https://avatars.githubusercontent.com/u/153510?v=4"},"body":"This introduces a performance testing framework under t/perf/.  It\ntries to be as close to the test-lib.sh infrastructure as possible,\nand thus should be easy to get used to for git developers.\n\nThe following points were considered for the implementation:\n\n1. You usually want to compare arbitrary revisions/build trees against\n   each other.  They may not have the performance test under\n   consideration, or even the perf-lib.sh infrastructure.\n\n   To cope with this, the 'run' script lets you specify arbitrary\n   build dirs and revisions.  It even automatically builds the revisions\n   if it doesn't have them at hand yet.\n\n2. Usually you would not want to run all tests.  It would take too\n   long anyway.  The 'run' script lets you specify which tests to run;\n   or you can also do it manually.  There is a Makefile for\n   discoverability and 'make clean', but it is not meant for\n   real-world use.\n\n3. Creating test repos from scratch in every test is extremely\n   time-consuming, and shipping or downloading such large/weird repos\n   is out of the question.\n\n   We leave this decision to the user.  Two different sizes of test\n   repos can be configured, and the scripts just copy one or more of\n   those (using hardlinks for the object store).  By default it tries\n   to use the build tree's git.git repository.\n\n   This is fairly fast and versatile.  Using a copy instead of a clone\n   preserves many properties that the user may want to test for, such\n   as lots of loose objects, unpacked refs, etc.\n\nSigned-off-by: Thomas Rast <trast@student.ethz.ch>\n---\n Makefile                        |   19 ++++-\n t/Makefile                      |    5 +-\n t/perf/.gitignore               |    2 +\n t/perf/Makefile                 |   15 +++\n t/perf/README                   |  146 ++++++++++++++++++++++++++++\n t/perf/aggregate.perl           |  166 ++++++++++++++++++++++++++++++++\n t/perf/min_time.perl            |   21 ++++\n t/perf/p0000-perf-lib-sanity.sh |   41 ++++++++\n t/perf/p0001-rev-list.sh        |   17 ++++\n t/perf/perf-lib.sh              |  199 +++++++++++++++++++++++++++++++++++++++\n t/perf/run                      |   82 ++++++++++++++++\n t/test-lib.sh                   |   12 ++-\n 12 files changed, 722 insertions(+), 3 deletions(-)\n create mode 100644 t/perf/.gitignore\n create mode 100644 t/perf/Makefile\n create mode 100644 t/perf/README\n create mode 100755 t/perf/aggregate.perl\n create mode 100755 t/perf/min_time.perl\n create mode 100755 t/perf/p0000-perf-lib-sanity.sh\n create mode 100755 t/perf/p0001-rev-list.sh\n create mode 100644 t/perf/perf-lib.sh\n create mode 100755 t/perf/run\n\ndiff --git a/Makefile b/Makefile\nindex 52506bf..c9e0a17 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -1769,7 +1769,7 @@ export DIFF TAR INSTALL DESTDIR SHELL_PATH\n \n SHELL = $(SHELL_PATH)\n \n-all:: shell_compatibility_test $(ALL_PROGRAMS) $(SCRIPT_LIB) $(BUILT_INS) $(OTHER_PROGRAMS) GIT-BUILD-OPTIONS\n+all:: shell_compatibility_test $(ALL_PROGRAMS) $(SCRIPT_LIB) $(BUILT_INS) $(OTHER_PROGRAMS) GIT-BUILD-OPTIONS GIT-TEST-OPTIONS\n ifneq (,$X)\n \t$(QUIET_BUILT_IN)$(foreach p,$(patsubst %$X,%,$(filter %$X,$(ALL_PROGRAMS) $(BUILT_INS) git$X)), test -d '$p' -o '$p' -ef '$p$X' || $(RM) '$p';)\n endif\n@@ -2228,6 +2228,18 @@ ifdef GIT_TEST_CMP_USE_COPIED_CONTEXT\n \t@echo GIT_TEST_CMP_USE_COPIED_CONTEXT=YesPlease >>$@\n endif\n \t@echo GETTEXT_POISON=\\''$(subst ','\\'',$(subst ','\\'',$(GETTEXT_POISON)))'\\' >>$@\n+ifdef GIT_PERF_REPEAT_COUNT\n+\t@echo GIT_PERF_REPEAT_COUNT=\\''$(subst ','\\'',$(subst ','\\'',$(GIT_PERF_REPEAT_COUNT)))'\\' >>$@\n+endif\n+ifdef GIT_PERF_REPO\n+\t@echo GIT_PERF_REPO=\\''$(subst ','\\'',$(subst ','\\'',$(GIT_PERF_REPO)))'\\' >>$@\n+endif\n+ifdef GIT_PERF_LARGE_REPO\n+\t@echo GIT_PERF_LARGE_REPO=\\''$(subst ','\\'',$(subst ','\\'',$(GIT_PERF_LARGE_REPO)))'\\' >>$@\n+endif\n+ifdef GIT_PERF_MAKE_OPTS\n+\t@echo GIT_PERF_MAKE_OPTS=\\''$(subst ','\\'',$(subst ','\\'',$(GIT_PERF_MAKE_OPTS)))'\\' >>$@\n+endif\n \n ### Detect Tck/Tk interpreter path changes\n ifndef NO_TCLTK\n@@ -2263,6 +2275,11 @@ export NO_SVN_TESTS\n test: all\n \t$(MAKE) -C t/ all\n \n+perf: all\n+\t$(MAKE) -C t/perf/ all\n+\n+.PHONY: test perf\n+\n test-ctype$X: ctype.o\n \n test-date$X: date.o ctype.o\ndiff --git a/t/Makefile b/t/Makefile\nindex 9046ec9..a481ab3 100644\n--- a/t/Makefile\n+++ b/t/Makefile\n@@ -73,6 +73,9 @@ gitweb-test:\n valgrind:\n \t$(MAKE) GIT_TEST_OPTS=\"$(GIT_TEST_OPTS) --valgrind\"\n \n+perf:\n+\t$(MAKE) -C perf/ all\n+\n # Smoke testing targets\n -include ../GIT-VERSION-FILE\n uname_S := $(shell sh -c 'uname -s 2>/dev/null || echo unknown')\n@@ -111,4 +114,4 @@ smoke_report: smoke\n \t\thttp://smoke.git.nix.is/app/projects/process_add_report/1 \\\n \t| grep -v ^Redirecting\n \n-.PHONY: pre-clean $(T) aggregate-results clean valgrind smoke smoke_report\n+.PHONY: pre-clean $(T) aggregate-results clean valgrind smoke smoke_report perf\ndiff --git a/t/perf/.gitignore b/t/perf/.gitignore\nnew file mode 100644\nindex 0000000..50f5cc1\n--- /dev/null\n+++ b/t/perf/.gitignore\n@@ -0,0 +1,2 @@\n+build/\n+test-results/\ndiff --git a/t/perf/Makefile b/t/perf/Makefile\nnew file mode 100644\nindex 0000000..8c47155\n--- /dev/null\n+++ b/t/perf/Makefile\n@@ -0,0 +1,15 @@\n+-include ../../config.mak\n+export GIT_TEST_OPTIONS\n+\n+all: perf\n+\n+perf: pre-clean\n+\t./run\n+\n+pre-clean:\n+\trm -rf test-results\n+\n+clean:\n+\trm -rf build \"trash directory\".* test-results\n+\n+.PHONY: all perf pre-clean clean\ndiff --git a/t/perf/README b/t/perf/README\nnew file mode 100644\nindex 0000000..fdf974a\n--- /dev/null\n+++ b/t/perf/README\n@@ -0,0 +1,146 @@\n+Git performance tests\n+=====================\n+\n+This directory holds performance testing scripts for git tools.  The\n+first part of this document describes the various ways in which you\n+can run them.\n+\n+When fixing the tools or adding enhancements, you are strongly\n+encouraged to add tests in this directory to cover what you are\n+trying to fix or enhance.  The later part of this short document\n+describes how your test scripts should be organized.\n+\n+\n+Running Tests\n+-------------\n+\n+The easiest way to run tests is to say \"make\".  This runs all\n+the tests on the current git repository.\n+\n+    === Running 2 tests in this tree ===\n+    [...]\n+    Test                                     this tree       \n+    ---------------------------------------------------------\n+    0001.1: rev-list --all                   0.54(0.51+0.02) \n+    0001.2: rev-list --all --objects         6.14(5.99+0.11) \n+    7810.1: grep worktree, cheap regex       0.16(0.16+0.35) \n+    7810.2: grep worktree, expensive regex   7.90(29.75+0.37)\n+    7810.3: grep --cached, cheap regex       3.07(3.02+0.25) \n+    7810.4: grep --cached, expensive regex   9.39(30.57+0.24)\n+\n+You can compare multiple repositories and even git revisions with the\n+'run' script:\n+\n+    $ ./run . origin/next /path/to/git-tree p0001-rev-list.sh\n+\n+where . stands for the current git tree.  The full invocation is\n+\n+    ./run [<revision|directory>...] [--] [<test-script>...]\n+\n+A '.' argument is implied if you do not pass any other\n+revisions/directories.\n+\n+You can also manually test this or another git build tree, and then\n+call the aggregation script to summarize the results:\n+\n+    $ ./p0001-rev-list.sh\n+    [...]\n+    $ GIT_BUILD_DIR=/path/to/other/git ./p0001-rev-list.sh\n+    [...]\n+    $ ./aggregate.perl . /path/to/other/git ./p0001-rev-list.sh\n+\n+aggregate.perl has the same invocation as 'run', it just does not run\n+anything beforehand.\n+\n+You can set the following variables (also in your config.mak):\n+\n+    GIT_PERF_REPEAT_COUNT\n+\tNumber of times a test should be repeated for best-of-N\n+\tmeasurements.  Defaults to 5.\n+\n+    GIT_PERF_MAKE_OPTS\n+\tOptions to use when automatically building a git tree for\n+\tperformance testing.  E.g., -j6 would be useful.\n+\n+    GIT_PERF_REPO\n+    GIT_PERF_LARGE_REPO\n+\tRepositories to copy for the performance tests.  The normal\n+\trepo should be at least git.git size.  The large repo should\n+\tprobably be about linux-2.6.git size for optimal results.\n+\tBoth default to the git.git you are running from.\n+\n+You can also pass the options taken by ordinary git tests; the most\n+useful one is:\n+\n+--root=<directory>::\n+\tCreate \"trash\" directories used to store all temporary data during\n+\ttesting under <directory>, instead of the t/ directory.\n+\tUsing this option with a RAM-based filesystem (such as tmpfs)\n+\tcan massively speed up the test suite.\n+\n+\n+Naming Tests\n+------------\n+\n+The performance test files are named as:\n+\n+\tpNNNN-commandname-details.sh\n+\n+where N is a decimal digit.  The same conventions for choosing NNNN as\n+for normal tests apply.\n+\n+\n+Writing Tests\n+-------------\n+\n+The perf script starts much like a normal test script, except it\n+sources perf-lib.sh:\n+\n+\t#!/bin/sh\n+\t#\n+\t# Copyright (c) 2005 Junio C Hamano\n+\t#\n+\n+\ttest_description='xxx performance test'\n+\t. ./perf-lib.sh\n+\n+After that you will want to use some of the following:\n+\n+\ttest_perf_default_repo  # sets up a \"normal\" repository\n+\ttest_perf_large_repo    # sets up a \"large\" repository\n+\n+\ttest_perf_default_repo sub  # ditto, in a subdir \"sub\"\n+\n+        test_checkout_worktree  # if you need the worktree too\n+\n+At least one of the first two is required!\n+\n+You can use test_expect_success as usual.  For actual performance\n+tests, use\n+\n+\ttest_perf 'descriptive string' '\n+\t\tcommand1 &&\n+\t\tcommand2\n+\t'\n+\n+test_perf spawns a subshell, for lack of better options.  This means\n+that\n+\n+* you _must_ export all variables that you need in the subshell\n+\n+* you _must_ flag all variables that you want to persist from the\n+  subshell with 'test_export':\n+\n+\ttest_perf 'descriptive string' '\n+\t\tfoo=$(git rev-parse HEAD) &&\n+\t\ttest_export foo\n+\t'\n+\n+  The so-exported variables are automatically marked for export in the\n+  shell executing the perf test.  For your convenience, test_export is\n+  the same as export in the main shell.\n+\n+  This feature relies on a bit of magic using 'set' and 'source'.\n+  While we have tried to make sure that it can cope with embedded\n+  whitespace and other special characters, it will not work with\n+  multi-line data.\ndiff --git a/t/perf/aggregate.perl b/t/perf/aggregate.perl\nnew file mode 100755\nindex 0000000..15f7fc1\n--- /dev/null\n+++ b/t/perf/aggregate.perl\n@@ -0,0 +1,166 @@\n+#!/usr/bin/perl\n+\n+use strict;\n+use warnings;\n+use Git;\n+\n+sub get_times {\n+\tmy $name = shift;\n+\topen my $fh, \"<\", $name or return undef;\n+\tmy $line = <$fh>;\n+\treturn undef if not defined $line;\n+\tclose $fh or die \"cannot close $name: $!\";\n+\t$line =~ /^(?:(\\d+):)?(\\d+):(\\d+(?:\\.\\d+)?) (\\d+(?:\\.\\d+)?) (\\d+(?:\\.\\d+)?)$/\n+\t\tor die \"bad input line: $line\";\n+\tmy $rt = ((defined $1 ? $1 : 0.0)*60+$2)*60+$3;\n+\treturn ($rt, $4, $5);\n+}\n+\n+sub format_times {\n+\tmy ($r, $u, $s, $firstr) = @_;\n+\tif (!defined $r) {\n+\t\treturn \"<missing>\";\n+\t}\n+\tmy $out = sprintf \"%.2f(%.2f+%.2f)\", $r, $u, $s;\n+\tif (defined $firstr) {\n+\t\tif ($firstr > 0) {\n+\t\t\t$out .= sprintf \" %+.1f%%\", 100.0*($r-$firstr)/$firstr;\n+\t\t} elsif ($r == 0) {\n+\t\t\t$out .= \" =\";\n+\t\t} else {\n+\t\t\t$out .= \" +inf\";\n+\t\t}\n+\t}\n+\treturn $out;\n+}\n+\n+my (@dirs, %dirnames, %dirabbrevs, %prefixes, @tests);\n+while (scalar @ARGV) {\n+\tmy $arg = $ARGV[0];\n+\tmy $dir;\n+\tlast if -f $arg or $arg eq \"--\";\n+\tif (! -d $arg) {\n+\t\tmy $rev = Git::command_oneline(qw(rev-parse --verify), $arg);\n+\t\t$dir = \"build/\".$rev;\n+\t} else {\n+\t\t$arg =~ s{/*$}{};\n+\t\t$dir = $arg;\n+\t\t$dirabbrevs{$dir} = $dir;\n+\t}\n+\tpush @dirs, $dir;\n+\t$dirnames{$dir} = $arg;\n+\tmy $prefix = $dir;\n+\t$prefix =~ tr/^a-zA-Z0-9/_/c;\n+\t$prefixes{$dir} = $prefix . '.';\n+\tshift @ARGV;\n+}\n+\n+if (not @dirs) {\n+\t@dirs = ('.');\n+}\n+$dirnames{'.'} = $dirabbrevs{'.'} = \"this tree\";\n+$prefixes{'.'} = '';\n+\n+shift @ARGV if scalar @ARGV and $ARGV[0] eq \"--\";\n+\n+@tests = @ARGV;\n+if (not @tests) {\n+\t@tests = glob \"p????-*.sh\";\n+}\n+\n+my @subtests;\n+my %shorttests;\n+for my $t (@tests) {\n+\t$t =~ s{(?:.*/)?(p(\\d+)-[^/]+)\\.sh$}{$1} or die \"bad test name: $t\";\n+\tmy $n = $2;\n+\tmy $fname = \"test-results/$t.subtests\";\n+\topen my $fp, \"<\", $fname or die \"cannot open $fname: $!\";\n+\tfor (<$fp>) {\n+\t\tchomp;\n+\t\t/^(\\d+)$/ or die \"malformed subtest line: $_\";\n+\t\tpush @subtests, \"$t.$1\";\n+\t\t$shorttests{\"$t.$1\"} = \"$n.$1\";\n+\t}\n+\tclose $fp or die \"cannot close $fname: $!\";\n+}\n+\n+sub read_descr {\n+\tmy $name = shift;\n+\topen my $fh, \"<\", $name or return \"<error reading description>\";\n+\tmy $line = <$fh>;\n+\tclose $fh or die \"cannot close $name\";\n+\tchomp $line;\n+\treturn $line;\n+}\n+\n+my %descrs;\n+my $descrlen = 4; # \"Test\"\n+for my $t (@subtests) {\n+\t$descrs{$t} = $shorttests{$t}.\": \".read_descr(\"test-results/$t.descr\");\n+\t$descrlen = length $descrs{$t} if length $descrs{$t}>$descrlen;\n+}\n+\n+sub have_duplicate {\n+\tmy %seen;\n+\tfor (@_) {\n+\t\treturn 1 if exists $seen{$_};\n+\t\t$seen{$_} = 1;\n+\t}\n+\treturn 0;\n+}\n+sub have_slash {\n+\tfor (@_) {\n+\t\treturn 1 if m{/};\n+\t}\n+\treturn 0;\n+}\n+\n+my %newdirabbrevs = %dirabbrevs;\n+while (!have_duplicate(values %newdirabbrevs)) {\n+\t%dirabbrevs = %newdirabbrevs;\n+\tlast if !have_slash(values %dirabbrevs);\n+\t%newdirabbrevs = %dirabbrevs;\n+\tfor (values %newdirabbrevs) {\n+\t\ts{^[^/]*/}{};\n+\t}\n+}\n+\n+my %times;\n+my @colwidth = ((0)x@dirs);\n+for my $i (0..$#dirs) {\n+\tmy $d = $dirs[$i];\n+\tmy $w = length (exists $dirabbrevs{$d} ? $dirabbrevs{$d} : $dirnames{$d});\n+\t$colwidth[$i] = $w if $w > $colwidth[$i];\n+}\n+for my $t (@subtests) {\n+\tmy $firstr;\n+\tfor my $i (0..$#dirs) {\n+\t\tmy $d = $dirs[$i];\n+\t\t$times{$prefixes{$d}.$t} = [get_times(\"test-results/$prefixes{$d}$t.times\")];\n+\t\tmy ($r,$u,$s) = @{$times{$prefixes{$d}.$t}};\n+\t\tmy $w = length format_times($r,$u,$s,$firstr);\n+\t\t$colwidth[$i] = $w if $w > $colwidth[$i];\n+\t\t$firstr = $r unless defined $firstr;\n+\t}\n+}\n+my $totalwidth = 3*@dirs+$descrlen;\n+$totalwidth += $_ for (@colwidth);\n+\n+printf \"%-${descrlen}s\", \"Test\";\n+for my $i (0..$#dirs) {\n+\tmy $d = $dirs[$i];\n+\tprintf \"   %-$colwidth[$i]s\", (exists $dirabbrevs{$d} ? $dirabbrevs{$d} : $dirnames{$d});\n+}\n+print \"\\n\";\n+print \"-\"x$totalwidth, \"\\n\";\n+for my $t (@subtests) {\n+\tprintf \"%-${descrlen}s\", $descrs{$t};\n+\tmy $firstr;\n+\tfor my $i (0..$#dirs) {\n+\t\tmy $d = $dirs[$i];\n+\t\tmy ($r,$u,$s) = @{$times{$prefixes{$d}.$t}};\n+\t\tprintf \"   %-$colwidth[$i]s\", format_times($r,$u,$s,$firstr);\n+\t\t$firstr = $r unless defined $firstr;\n+\t}\n+\tprint \"\\n\";\n+}\ndiff --git a/t/perf/min_time.perl b/t/perf/min_time.perl\nnew file mode 100755\nindex 0000000..c1a2717\n--- /dev/null\n+++ b/t/perf/min_time.perl\n@@ -0,0 +1,21 @@\n+#!/usr/bin/perl\n+\n+my $minrt = 1e100;\n+my $min;\n+\n+while (<>) {\n+\t# [h:]m:s.xx U.xx S.xx\n+\t/^(?:(\\d+):)?(\\d+):(\\d+(?:\\.\\d+)?) (\\d+(?:\\.\\d+)?) (\\d+(?:\\.\\d+)?)$/\n+\t\tor die \"bad input line: $_\";\n+\tmy $rt = ((defined $1 ? $1 : 0.0)*60+$2)*60+$3;\n+\tif ($rt < $minrt) {\n+\t\t$min = $_;\n+\t\t$minrt = $rt;\n+\t}\n+}\n+\n+if (!defined $min) {\n+\tdie \"no input found\";\n+}\n+\n+print $min;\ndiff --git a/t/perf/p0000-perf-lib-sanity.sh b/t/perf/p0000-perf-lib-sanity.sh\nnew file mode 100755\nindex 0000000..2ca4aac\n--- /dev/null\n+++ b/t/perf/p0000-perf-lib-sanity.sh\n@@ -0,0 +1,41 @@\n+#!/bin/sh\n+\n+test_description='Tests whether perf-lib facilities work'\n+. ./perf-lib.sh\n+\n+test_perf_default_repo\n+\n+test_perf 'test_perf_default_repo works' '\n+\tfoo=$(git rev-parse HEAD) &&\n+\ttest_export foo\n+'\n+\n+test_checkout_worktree\n+\n+test_perf 'test_checkout_worktree works' '\n+\twt=$(find . | wc -l) &&\n+\tidx=$(git ls-files | wc -l) &&\n+\ttest $wt -gt $idx\n+'\n+\n+baz=baz\n+test_export baz\n+\n+test_expect_success 'test_export works' '\n+\techo \"$foo\" &&\n+\ttest \"$foo\" = \"$(git rev-parse HEAD)\" &&\n+\techo \"$baz\" &&\n+\ttest \"$baz\" = baz\n+'\n+\n+test_perf 'export a weird var' '\n+\tbar=\"weird # variable\" &&\n+\ttest_export bar\n+'\n+\n+test_expect_success 'test_export works with weird vars' '\n+\techo \"$bar\" &&\n+\ttest \"$bar\" = \"weird # variable\"\n+'\n+\n+test_done\ndiff --git a/t/perf/p0001-rev-list.sh b/t/perf/p0001-rev-list.sh\nnew file mode 100755\nindex 0000000..4f71a63\n--- /dev/null\n+++ b/t/perf/p0001-rev-list.sh\n@@ -0,0 +1,17 @@\n+#!/bin/sh\n+\n+test_description=\"Tests history walking performance\"\n+\n+. ./perf-lib.sh\n+\n+test_perf_default_repo\n+\n+test_perf 'rev-list --all' '\n+\tgit rev-list --all >/dev/null\n+'\n+\n+test_perf 'rev-list --all --objects' '\n+\tgit rev-list --all --objects >/dev/null\n+'\n+\n+test_done\ndiff --git a/t/perf/perf-lib.sh b/t/perf/perf-lib.sh\nnew file mode 100644\nindex 0000000..ee6c083\n--- /dev/null\n+++ b/t/perf/perf-lib.sh\n@@ -0,0 +1,199 @@\n+#!/bin/bash\n+#\n+# Copyright (c) 2011 Thomas Rast\n+#\n+# This program is free software: you can redistribute it and/or modify\n+# it under the terms of the GNU General Public License as published by\n+# the Free Software Foundation, either version 2 of the License, or\n+# (at your option) any later version.\n+#\n+# This program is distributed in the hope that it will be useful,\n+# but WITHOUT ANY WARRANTY; without even the implied warranty of\n+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the\n+# GNU General Public License for more details.\n+#\n+# You should have received a copy of the GNU General Public License\n+# along with this program.  If not, see http://www.gnu.org/licenses/ .\n+\n+# do the --tee work early; it otherwise confuses our careful\n+# GIT_BUILD_DIR mangling\n+case \"$GIT_TEST_TEE_STARTED, $* \" in\n+done,*)\n+\t# do not redirect again\n+\t;;\n+*' --tee '*|*' --va'*)\n+\tmkdir -p test-results\n+\tBASE=test-results/$(basename \"$0\" .sh)\n+\t(GIT_TEST_TEE_STARTED=done ${SHELL-sh} \"$0\" \"$@\" 2>&1;\n+\t echo $? > $BASE.exit) | tee $BASE.out\n+\ttest \"$(cat $BASE.exit)\" = 0\n+\texit\n+\t;;\n+esac\n+\n+TEST_DIRECTORY=$(pwd)/..\n+TEST_OUTPUT_DIRECTORY=$(pwd)\n+if test -z \"$GIT_BUILD_DIR\"; then\n+\tGIT_BUILD_DIR=$TEST_DIRECTORY/..\n+\tperf_results_prefix=\n+else\n+\tperf_results_prefix=$(printf \"%s\" \"$GIT_BUILD_DIR\" | tr -c \"[a-zA-Z0-9]\" \"[_*]\")\".\"\n+\t# make the build dir absolute\n+\tGIT_BUILD_DIR=$(cd \"$GIT_BUILD_DIR\" && pwd)\n+fi\n+\n+TEST_NO_CREATE_REPO=t\n+\n+. ../test-lib.sh\n+\n+perf_results_dir=$TEST_OUTPUT_DIRECTORY/test-results\n+mkdir -p \"$perf_results_dir\"\n+rm -f \"$perf_results_dir\"/$(basename \"$0\" .sh).subtests\n+\n+if test -z \"$GIT_PERF_REPEAT_COUNT\"; then\n+\tGIT_PERF_REPEAT_COUNT=3\n+fi\n+die_if_build_dir_not_repo () {\n+\tif ! ( cd \"$TEST_DIRECTORY/..\" &&\n+\t\t    git rev-parse --build-dir >/dev/null 2>&1 ); then\n+\t\terror \"No $1 defined, and your build directory is not a repo\"\n+\tfi\n+}\n+\n+if test -z \"$GIT_PERF_REPO\"; then\n+\tdie_if_build_dir_not_repo '$GIT_PERF_REPO'\n+\tGIT_PERF_REPO=$TEST_DIRECTORY/..\n+fi\n+if test -z \"$GIT_PERF_LARGE_REPO\"; then\n+\tdie_if_build_dir_not_repo '$GIT_PERF_LARGE_REPO'\n+\tGIT_PERF_LARGE_REPO=$TEST_DIRECTORY/..\n+fi\n+\n+test_perf_create_repo_from () {\n+\ttest \"$#\" = 2 ||\n+\terror \"bug in the test script: not 2 parameters to test-create-repo\"\n+\trepo=\"$1\"\n+\tsource=\"$2\"\n+\tsource_git=$source/$(cd \"$source\" && git rev-parse --git-dir)\n+\tmkdir -p \"$repo/.git\"\n+\t(\n+\t\tcd \"$repo/.git\" &&\n+\t\t{ cp -Rl \"$source_git/objects\" . 2>/dev/null ||\n+\t\t\tcp -R \"$source_git/objects\" .; } &&\n+\t\tfor stuff in \"$source_git\"/*; do\n+\t\t\tcase \"$stuff\" in\n+\t\t\t\t*/objects|*/hooks|*/config)\n+\t\t\t\t\t;;\n+\t\t\t\t*)\n+\t\t\t\t\tcp -R \"$stuff\" . || break\n+\t\t\t\t\t;;\n+\t\t\tesac\n+\t\tdone &&\n+\t\tcd .. &&\n+\t\tgit init -q &&\n+\t\tmv .git/hooks .git/hooks-disabled 2>/dev/null\n+\t) || error \"failed to copy repository '$source' to '$repo'\"\n+}\n+\n+# call at least one of these to establish an appropriately-sized repository\n+test_perf_default_repo () {\n+\ttest_perf_create_repo_from \"${1:-$TRASH_DIRECTORY}\" \"$GIT_PERF_REPO\"\n+}\n+test_perf_large_repo () {\n+\tif test \"$GIT_PERF_LARGE_REPO\" = \"$GIT_BUILD_DIR\"; then\n+\t\techo \"warning: \\$GIT_PERF_LARGE_REPO is \\$GIT_BUILD_DIR.\" >&2\n+\t\techo \"warning: This will work, but may not be a sufficiently large repo\" >&2\n+\t\techo \"warning: for representative measurements.\" >&2\n+\tfi\n+\ttest_perf_create_repo_from \"${1:-$TRASH_DIRECTORY}\" \"$GIT_PERF_LARGE_REPO\"\n+}\n+test_checkout_worktree () {\n+\tgit checkout-index -u -a ||\n+\terror \"git checkout-index failed\"\n+}\n+\n+# Performance tests should never fail.  If they do, stop immediately\n+immediate=t\n+\n+test_run_perf_ () {\n+\ttest_cleanup=:\n+\ttest_export_=\"test_cleanup\"\n+\texport test_cleanup test_export_\n+\t/usr/bin/time -f \"%E %U %S\" -o test_time.$i \"$SHELL\" -c '\n+. '\"$TEST_DIRECTORY\"/../test-lib-functions.sh'\n+test_export () {\n+\t[ $# != 0 ] || return 0\n+\ttest_export_=\"$test_export_\\\\|$1\"\n+\tshift\n+\ttest_export \"$@\"\n+}\n+'\"$1\"'\n+ret=$?\n+set | sed -n \"s'\"/'/'\\\\\\\\''/g\"';s/^\\\\($test_export_\\\\)/export '\"'&'\"'/p\" >test_vars\n+exit $ret' >&3 2>&4\n+\teval_ret=$?\n+\n+\tif test $eval_ret = 0 || test -n \"$expecting_failure\"\n+\tthen\n+\t\ttest_eval_ \"$test_cleanup\"\n+\t\tsource ./test_vars || error \"failed to load updated environment\"\n+\tfi\n+\tif test \"$verbose\" = \"t\" && test -n \"$HARNESS_ACTIVE\"; then\n+\t\techo \"\"\n+\tfi\n+\treturn \"$eval_ret\"\n+}\n+\n+\n+test_perf () {\n+\ttest \"$#\" = 3 && { test_prereq=$1; shift; } || test_prereq=\n+\ttest \"$#\" = 2 ||\n+\terror \"bug in the test script: not 2 or 3 parameters to test-expect-success\"\n+\texport test_prereq\n+\tif ! test_skip \"$@\"\n+\tthen\n+\t\tbase=$(basename \"$0\" .sh)\n+\t\techo \"$test_count\" >>\"$perf_results_dir\"/$base.subtests\n+\t\techo \"$1\" >\"$perf_results_dir\"/$base.descr\n+\t\tif test -z \"$verbose\"; then\n+\t\t\techo -n \"perf $test_count - $1:\"\n+\t\telse\n+\t\t\techo \"perf $test_count - $1:\"\n+\t\tfi\n+\t\tfor i in $(seq 1 $GIT_PERF_REPEAT_COUNT); do\n+\t\t\tsay >&3 \"running: $2\"\n+\t\t\tif test_run_perf_ \"$2\"\n+\t\t\tthen\n+\t\t\t\tif test -z \"$verbose\"; then\n+\t\t\t\t\techo -n \" $i\"\n+\t\t\t\telse\n+\t\t\t\t\techo \"* timing run $i/$GIT_PERF_REPEAT_COUNT:\"\n+\t\t\t\tfi\n+\t\t\telse\n+\t\t\t\ttest -z \"$verbose\" && echo\n+\t\t\t\ttest_failure_ \"$@\"\n+\t\t\t\tbreak\n+\t\t\tfi\n+\t\tdone\n+\t\tif test -z \"$verbose\"; then\n+\t\t\techo \" ok\"\n+\t\telse\n+\t\t\ttest_ok_ \"$1\"\n+\t\tfi\n+\t\tbase=\"$perf_results_dir\"/\"$perf_results_prefix$(basename \"$0\" .sh)\".\"$test_count\"\n+\t\t\"$TEST_DIRECTORY\"/perf/min_time.perl test_time.* >\"$base\".times\n+\tfi\n+\techo >&3 \"\"\n+}\n+\n+# We extend test_done to print timings at the end (./run disables this\n+# and does it after running everything)\n+test_at_end_hook_ () {\n+\tif test -z \"$GIT_PERF_AGGREGATING_LATER\"; then\n+\t\t( cd \"$TEST_DIRECTORY\"/perf && ./aggregate.perl $(basename \"$0\") )\n+\tfi\n+}\n+\n+test_export () {\n+\texport \"$@\"\n+}\ndiff --git a/t/perf/run b/t/perf/run\nnew file mode 100755\nindex 0000000..85ead88\n--- /dev/null\n+++ b/t/perf/run\n@@ -0,0 +1,82 @@\n+#!/bin/sh\n+\n+case \"$1\" in\n+\t--help)\n+\t\techo \"usage: $0 [other_git_tree...] [--] [test_scripts]\"\n+\t\texit 0\n+\t\t;;\n+esac\n+\n+die () {\n+\techo >&2 \"error: $*\"\n+\texit 1\n+}\n+\n+run_one_dir () {\n+\tif test $# -eq 0; then\n+\t\tset -- p????-*.sh\n+\tfi\n+\techo \"=== Running $# tests in ${GIT_BUILD_DIR:-this tree} ===\"\n+\tfor t in \"$@\"; do\n+\t\t./$t $GIT_TEST_OPTS\n+\tdone\n+}\n+\n+unpack_git_rev () {\n+\trev=$1\n+\tmkdir -p build/$rev\n+\t(cd \"$(git rev-parse --show-cdup)\" && git archive --format=tar $rev) |\n+\t(cd build/$rev && tar x)\n+}\n+build_git_rev () {\n+\trev=$1\n+\tcp ../../config.mak build/$rev/config.mak\n+\t(cd build/$rev && make $GIT_PERF_MAKE_OPTS) ||\n+\tdie \"failed to build revision '$mydir'\"\n+}\n+\n+run_dirs_helper () {\n+\tmydir=${1%/}\n+\tshift\n+\twhile test $# -gt 0 -a \"$1\" != -- -a ! -f \"$1\"; do\n+\t\tshift\n+\tdone\n+\tif test $# -gt 0 -a \"$1\" == --; then\n+\t\tshift\n+\tfi\n+\tif [ ! -d \"$mydir\" ]; then\n+\t\trev=$(git rev-parse --verify \"$mydir\" 2>/dev/null) ||\n+\t\tdie \"'$mydir' is neither a directory nor a valid revision\"\n+\t\tif [ ! -d build/$rev ]; then\n+\t\t\tunpack_git_rev $rev\n+\t\tfi\n+\t\tbuild_git_rev $rev\n+\t\tmydir=build/$rev\n+\tfi\n+\tif test \"$mydir\" = .; then\n+\t\tunset GIT_BUILD_DIR\n+\telse\n+\t\tGIT_BUILD_DIR=\"$mydir\"\n+\t\texport GIT_BUILD_DIR\n+\tfi\n+\trun_one_dir \"$@\"\n+}\n+\n+run_dirs () {\n+\twhile test $# -gt 0 -a \"$1\" != -- -a ! -f \"$1\"; do\n+\t\trun_dirs_helper \"$@\"\n+\t\tshift\n+\tdone\n+}\n+\n+GIT_PERF_AGGREGATING_LATER=t\n+export GIT_PERF_AGGREGATING_LATER\n+\n+cd \"$(dirname $0)\"\n+. ../../GIT-TEST-OPTIONS\n+\n+if test $# = 0 -o \"$1\" = -- -o -f \"$1\"; then\n+\tset -- . \"$@\"\n+fi\n+run_dirs \"$@\"\n+./aggregate.perl \"$@\"\ndiff --git a/t/test-lib.sh b/t/test-lib.sh\nindex 06f8c96..83bb7b3 100644\n--- a/t/test-lib.sh\n+++ b/t/test-lib.sh\n@@ -55,6 +55,7 @@ unset $(perl -e '\n \t\tPROVE\n \t\tVALGRIND\n \t\tBUILD_DIR\n+\t\tPERF_AGGREGATING_LATER\n \t));\n \tmy @vars = grep(/^GIT_/ && !/^GIT_($ok)/o, @env);\n \tprint join(\"\\n\", @vars);\n@@ -313,6 +314,9 @@ test_skip () {\n \tesac\n }\n \n+# stub; perf-lib overrides it\n+test_at_end_hook_ () {\n+\t:\n }\n \n test_done () {\n@@ -358,6 +362,8 @@ test_done () {\n \t\tcd \"$(dirname \"$remove_trash\")\" &&\n \t\trm -rf \"$(basename \"$remove_trash\")\"\n \n+\t\ttest_at_end_hook_\n+\n \t\texit 0 ;;\n \n \t*)\n@@ -539,7 +545,11 @@ rm -fr \"$test\" || {\n HOME=\"$TRASH_DIRECTORY\"\n export HOME\n \n-test_create_repo \"$test\"\n+if test -z \"$TEST_NO_CREATE_REPO\"; then\n+\ttest_create_repo \"$test\"\n+else\n+\tmkdir -p \"$test\"\n+fi\n # Use -P to resolve symlinks in our working directory so that the cwd\n # in subprocesses like git equals our $PWD (for pathname comparisons).\n cd -P \"$test\" || exit 1\n-- \n1.7.8.304.ge42e4\n"},{"id":"181149","messageId":"370de2c5c277513098480baf47f73ca6077787b0.1323876121.git.trast@student.ethz.ch","threadId":"29165","inReplyTo":"cover.1323876121.git.trast@student.ethz.ch","subject":"[RFC PATCH 4/4] Add a performance test for git-grep","fromName":"Thomas Rast","fromEmail":"trast@student.ethz.ch","sentAt":"2011-12-14T15:23:32Z","receivedAt":"2011-12-14T15:23:32Z","isPatch":true,"sender":{"key":"tr@thomasrast.ch","avatar":"https://avatars.githubusercontent.com/u/153510?v=4"},"body":"The only catch is that we don't really know what our repo contains, so\nwe have to ignore any possible \"not found\" status from git-grep.\n\nSigned-off-by: Thomas Rast <trast@student.ethz.ch>\n---\n t/perf/p7810-grep.sh |   23 +++++++++++++++++++++++\n 1 files changed, 23 insertions(+), 0 deletions(-)\n create mode 100755 t/perf/p7810-grep.sh\n\ndiff --git a/t/perf/p7810-grep.sh b/t/perf/p7810-grep.sh\nnew file mode 100755\nindex 0000000..9f4ade6\n--- /dev/null\n+++ b/t/perf/p7810-grep.sh\n@@ -0,0 +1,23 @@\n+#!/bin/sh\n+\n+test_description=\"git-grep performance in various modes\"\n+\n+. ./perf-lib.sh\n+\n+test_perf_large_repo\n+test_checkout_worktree\n+\n+test_perf 'grep worktree, cheap regex' '\n+\tgit grep some_nonexistent_string || :\n+'\n+test_perf 'grep worktree, expensive regex' '\n+\tgit grep \"^.* *some_nonexistent_string$\" || :\n+'\n+test_perf 'grep --cached, cheap regex' '\n+\tgit grep --cached some_nonexistent_string || :\n+'\n+test_perf 'grep --cached, expensive regex' '\n+\tgit grep --cached \"^.* *some_nonexistent_string$\" || :\n+'\n+\n+test_done\n-- \n1.7.8.304.ge42e4\n"},{"id":"181155","messageId":"201112141704.32052.trast@student.ethz.ch","threadId":"29165","inReplyTo":"91630874c45015e74cf72df03b6011856fd0fe4f.1323876121.git.trast@student.ethz.ch","subject":"Re: [RFC PATCH 3/4] Introduce a performance testing framework","fromName":"Thomas Rast","fromEmail":"trast@student.ethz.ch","sentAt":"2011-12-14T16:04:31Z","receivedAt":"2011-12-14T16:04:31Z","isPatch":true,"sender":{"key":"tr@thomasrast.ch","avatar":"https://avatars.githubusercontent.com/u/153510?v=4"},"body":"This patch actually needs the below fixup on top.  Old files from\nprevious runs had fooled my testing...\n\ndiff --git a/t/perf/perf-lib.sh b/t/perf/perf-lib.sh\nindex ee6c083..f2a0b73 100644\n--- a/t/perf/perf-lib.sh\n+++ b/t/perf/perf-lib.sh\n@@ -154,7 +154,7 @@ test_perf () {\n \tthen\n \t\tbase=$(basename \"$0\" .sh)\n \t\techo \"$test_count\" >>\"$perf_results_dir\"/$base.subtests\n-\t\techo \"$1\" >\"$perf_results_dir\"/$base.descr\n+\t\techo \"$1\" >\"$perf_results_dir\"/$base.$test_count.descr\n \t\tif test -z \"$verbose\"; then\n \t\t\techo -n \"perf $test_count - $1:\"\n \t\telse\n\n-- \nThomas Rast\ntrast@{inf,student}.ethz.ch\n"},{"id":"181202","messageId":"7vsjkmpm6g.fsf@alter.siamese.dyndns.org","threadId":"29165","inReplyTo":"94f64a03398829bb9a11c18577efb39d9b153eca.1323876121.git.trast@student.ethz.ch","subject":"Re: [RFC PATCH 2/4] test-lib: allow testing another git build tree","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2011-12-15T03:07:35Z","receivedAt":"2011-12-15T03:07:35Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Thomas Rast <trast@student.ethz.ch> writes:\n\n> The perf-lib work wants this feature, so we may as well do it for\n> test-lib in general.\n\nHow is this different from what GIT_TEST_INSTALLED already gives us\n(other than \"needs more diskspace to keep another source tree fully\nbuilt\", that is)?\n"},{"id":"181222","messageId":"3016269.uH93UWUbx5@thomas.inf.ethz.ch","threadId":"29165","inReplyTo":"7vsjkmpm6g.fsf@alter.siamese.dyndns.org","subject":"Re: [RFC PATCH 2/4] test-lib: allow testing another git build tree","fromName":"Thomas Rast","fromEmail":"trast@student.ethz.ch","sentAt":"2011-12-15T10:33:45Z","receivedAt":"2011-12-15T10:33:45Z","isPatch":true,"sender":{"key":"tr@thomasrast.ch","avatar":"https://avatars.githubusercontent.com/u/153510?v=4"},"body":"On Wednesday 14 December 2011 19:07:35 Junio C Hamano wrote:\n> Thomas Rast <trast@student.ethz.ch> writes:\n> > The perf-lib work wants this feature, so we may as well do it for\n> > test-lib in general.\n> \n> How is this different from what GIT_TEST_INSTALLED already gives us\n> (other than \"needs more diskspace to keep another source tree fully\n> built\", that is)?\n\nI was scared away by the note that it would use (among others) perl\nlibs from the current build tree.  Upon investigation I also see that\nthe test-* situation is still not satisfactory.  Some (like\ntest-chmtime) are used by the tests for a vital task, and if they ever\nhave to be fixed, we would want to use the fixed version in any \"test\nan old git\" run.  OTOH, others (e.g., test-dump-cache-tree) are linked\nwith the rest of the code and serve to test an otherwise not\naccessible part of it, and testing an old git should use them from the\ntested tree.\n\nThe disk space argument is moot IMO: for sane perf testing you need\nthe extra build tree anyway because you cannot checkout another\nversion in the current tree.  Otherwise the scripts may change and/or\ndisappear from under themselves.\n\nAn optimization might be to have the run script only use a single\nbuild tree and several install trees.  However, while such a built\ntree takes just over 100MB of space in my tests, everything installed\n$PREFIX/libexec/git-core is also already 65MB here.  So the latter\nscheme would only amortize itself if you had at least 3 trees tested\nsimultaneously.\n\n-- \nThomas Rast\ntrast@{inf,student}.ethz.ch\n"},{"id":"181241","messageId":"7vaa6toduq.fsf@alter.siamese.dyndns.org","threadId":"29165","inReplyTo":"3016269.uH93UWUbx5@thomas.inf.ethz.ch","subject":"Re: [RFC PATCH 2/4] test-lib: allow testing another git build tree","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2011-12-15T19:05:01Z","receivedAt":"2011-12-15T19:05:01Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Thomas Rast <trast@student.ethz.ch> writes:\n\n> On Wednesday 14 December 2011 19:07:35 Junio C Hamano wrote:\n>> Thomas Rast <trast@student.ethz.ch> writes:\n>> > The perf-lib work wants this feature, so we may as well do it for\n>> > test-lib in general.\n>> \n>> How is this different from what GIT_TEST_INSTALLED already gives us\n>> (other than \"needs more diskspace to keep another source tree fully\n>> built\", that is)?\n>\n> I was scared away by the note that it would use (among others) perl\n> libs from the current build tree.  Upon investigation I also see that\n> the test-* situation is still not satisfactory.  Some (like\n> test-chmtime) are used by the tests for a vital task, and if they ever\n> have to be fixed, we would want to use the fixed version in any \"test\n> an old git\" run.  OTOH, others (e.g., test-dump-cache-tree) are linked\n> with the rest of the code and serve to test an otherwise not\n> accessible part of it, and testing an old git should use them from the\n> tested tree.\n\nOkay, so GIT_BUILD_DIR variant should now use everything from the tree\nbeing tested for consistency to improve the situation, right?\n\nEhh, not quite, if you worry about an old one having broken test-chmtime,\nyou actively do not want that consistency; how is that addressed?\n\nNot that I think we should address such issues---these standalone programs\ntend to be small and simple, so always using them from the tree being\ntested is a much better policy than using them from the tree you are\nrunning the tests from.\n\n> The disk space argument is moot IMO:\n\nI was not really worried about \"space\" (disks are cheap) but more about\nmental burden of having to keep multiple build trees in the source area.\nKeeping multiple installed versions is already necessary to be able to\nquickly respond to \"I am using Git version X and this does not work\".\n"}]}