[RFC PATCH 0/4] Performance test framework

Hi,

This is the first shot at a usable performance test suite. It's another angle than the refperf work of Michael Haggerty, however he helped shape this in #git-devel (thanks!).

There's a big README, but if you just want to dive in, here's me comparing the grep performance of my POC "pack reading in parallel" branch against Junio's next:

  $ cd t/perf
  $ ./run origin/next t/sha1_file-parallel p7810-grep.sh 
  [snip no-op compilation]
  === Running 1 tests in build/7a6d658aa3c9016dd04ff3515cbf15edca6562a4 ===
  perf 1 - grep worktree, cheap regex: 1 2 3 4 5 ok
  perf 2 - grep worktree, expensive regex: 1 2 3 4 5 ok
  perf 3 - grep --cached, cheap regex: 1 2 3 4 5 ok
  perf 4 - grep --cached, expensive regex: 1 2 3 4 5 ok
  # passed all 4 test(s)
  1..4
  GIT_VERSION = 1.7.8.GIT
      * new build flags or prefix
      * new link flags
      GEN common-cmds.h
      CC hex.o
      CC kwset.o
  [snip rest of compilation]
  === Running 1 tests in build/dd2bf650b382f5aca727b7d93a48598fb1a2f7d9 ===
  perf 1 - grep worktree, cheap regex: 1 2 3 4 5 ok
  perf 2 - grep worktree, expensive regex: 1 2 3 4 5 ok
  perf 3 - grep --cached, cheap regex: 1 2 3 4 5 ok
  perf 4 - grep --cached, expensive regex: 1 2 3 4 5 ok
  # passed all 4 test(s)
  1..4
  Test                                     origin/next        t/sha1_file-parallel  
  ----------------------------------------------------------------------------------
  7810.1: grep worktree, cheap regex       0.16(0.16+0.35)    0.16(0.15+0.36) +0.0% 
  7810.2: grep worktree, expensive regex   7.83(29.68+0.39)   7.95(29.98+0.39) +1.5%
  7810.3: grep --cached, cheap regex       3.12(3.11+0.24)    1.11(3.46+0.18) -64.4%
  7810.4: grep --cached, expensive regex   9.43(30.53+0.28)   8.89(32.99+0.22) -5.7%

Note in particular that neither of the two branches contains the perf work.

Have fun!

Thomas Rast (4):
  Move the user-facing test library to test-lib-functions.sh
  test-lib: allow testing another git build tree
  Introduce a performance testing framework
  Add a performance test for git-grep

 Makefile                        |   26 ++-
 t/Makefile                      |    5 +-
 t/perf/.gitignore               |    2 +
 t/perf/Makefile                 |   15 +
 t/perf/README                   |  146 ++++++++++
 t/perf/aggregate.perl           |  166 ++++++++++++
 t/perf/min_time.perl            |   21 ++
 t/perf/p0000-perf-lib-sanity.sh |   41 +++
 t/perf/p0001-rev-list.sh        |   17 ++
 t/perf/p7810-grep.sh            |   23 ++
 t/perf/perf-lib.sh              |  199 ++++++++++++++
 t/perf/run                      |   82 ++++++
 t/test-lib-functions.sh         |  528 ++++++++++++++++++++++++++++++++++++
 t/test-lib.sh                   |  561 +++------------------------------------
 14 files changed, 1300 insertions(+), 532 deletions(-)
 create mode 100644 t/perf/.gitignore
 create mode 100644 t/perf/Makefile
 create mode 100644 t/perf/README
 create mode 100755 t/perf/aggregate.perl
 create mode 100755 t/perf/min_time.perl
 create mode 100755 t/perf/p0000-perf-lib-sanity.sh
 create mode 100755 t/perf/p0001-rev-list.sh
 create mode 100755 t/perf/p7810-grep.sh
 create mode 100644 t/perf/perf-lib.sh
 create mode 100755 t/perf/run
 create mode 100644 t/test-lib-functions.sh