{"thread":{"id":"29918","subject":"[PATCH 09/11] perf/run: allow skipping some revisions","startedAt":"2012-03-12T15:09:56Z","lastAt":"2012-03-12T20:59:51Z","messageCount":20,"participants":["Thomas Rast","Nguyen Thai Ngoc Duy","Jakub Narebski","Junio C Hamano"],"isPatch":true,"patchVersion":1,"patchTotal":11},"messages":[{"id":"186722","messageId":"cover.1331561353.git.trast@student.ethz.ch","threadId":"29918","inReplyTo":null,"subject":"[PATCH 00/11] perf improvements past v1.7.10","fromName":"Thomas Rast","fromEmail":"trast@student.ethz.ch","sentAt":"2012-03-12T15:09:56Z","receivedAt":"2012-03-12T15:09:56Z","isPatch":true,"sender":{"key":"tr@thomasrast.ch","avatar":"https://avatars.githubusercontent.com/u/153510?v=4"},"body":"This is what I have collected over a week of playing with the perf\nsuite.  There is no rush; I am just sending them out for anyone\ninterested.  If you plan on using the perf suite, I would still\nsuggest you apply them.\n\nAside from the new tests, the main goal is the bisection script.\n[8/11] is fun too.  Try it!  You will have to install GNU R however.\n\nThomas Rast (11):\n  perf/aggregate: load Git.pm from the build tree\n  Introduce a performance test for git-rebase\n  Introduce a performance test for git-blame\n  perf: display average instead of minimum time\n  perf: suppress aggregation also in 'run'\n  perf: dereference to a commit when building\n  perf: convert realtime to seconds when collecting runs\n  perf/aggregate: optionally include a t-test score\n  perf/run: allow skipping some revisions\n  perf: implement a test-selection feature\n  perf: add a bisection tool\n\n t/perf/README          |    8 +++--\n t/perf/aggregate.perl  |   65 ++++++++++++++++++++++++++--------\n t/perf/bisect_slowdown |   88 ++++++++++++++++++++++++++++++++++++++++++++++\n t/perf/min_time.perl   |   21 -----------\n t/perf/p3400-rebase.sh |   91 ++++++++++++++++++++++++++++++++++++++++++++++++\n t/perf/p8002-blame.sh  |   46 ++++++++++++++++++++++++\n t/perf/perf-lib.sh     |   12 +++++--\n t/perf/run             |   10 ++++--\n t/perf/t_test_score.sh |   24 +++++++++++++\n 9 files changed, 322 insertions(+), 43 deletions(-)\n create mode 100755 t/perf/bisect_slowdown\n delete mode 100755 t/perf/min_time.perl\n create mode 100755 t/perf/p3400-rebase.sh\n create mode 100755 t/perf/p8002-blame.sh\n create mode 100755 t/perf/t_test_score.sh\n\n-- \n1.7.10.rc0.230.g16d90\n"},{"id":"186721","messageId":"1f991c7d82e1f55680c7b43790fbe37bdd686415.1331561353.git.trast@student.ethz.ch","threadId":"29918","inReplyTo":"cover.1331561353.git.trast@student.ethz.ch","subject":"[PATCH 01/11] perf/aggregate: load Git.pm from the build tree","fromName":"Thomas Rast","fromEmail":"trast@student.ethz.ch","sentAt":"2012-03-12T15:09:57Z","receivedAt":"2012-03-12T15:09:57Z","isPatch":true,"sender":{"key":"tr@thomasrast.ch","avatar":"https://avatars.githubusercontent.com/u/153510?v=4"},"body":"As pointed out by Junio, aggregate.perl is not careful enough about\nits loading of Git.pm, using the one that is installed on the system.\nLoad the version from the build tree, using FindBin as proposed by\nJakub.\n\nSigned-off-by: Thomas Rast <trast@student.ethz.ch>\n---\n t/perf/aggregate.perl |    3 +++\n 1 file changed, 3 insertions(+)\n\ndiff --git a/t/perf/aggregate.perl b/t/perf/aggregate.perl\nindex 15f7fc1..c0afa0b 100755\n--- a/t/perf/aggregate.perl\n+++ b/t/perf/aggregate.perl\n@@ -2,6 +2,9 @@\n \n use strict;\n use warnings;\n+use FindBin;\n+use lib \"$FindBin::Bin/../../perl/blib/lib\",\n+\t\"$FindBin::Bin/../../perl/blib/arch/auto/Git\";\n use Git;\n \n sub get_times {\n-- \n1.7.10.rc0.230.g16d90\n"},{"id":"186716","messageId":"e6a6af53264368a25e71719901f3227042e21f5c.1331561353.git.trast@student.ethz.ch","threadId":"29918","inReplyTo":"cover.1331561353.git.trast@student.ethz.ch","subject":"[PATCH 02/11] Introduce a performance test for git-rebase","fromName":"Thomas Rast","fromEmail":"trast@student.ethz.ch","sentAt":"2012-03-12T15:09:58Z","receivedAt":"2012-03-12T15:09:58Z","isPatch":true,"sender":{"key":"tr@thomasrast.ch","avatar":"https://avatars.githubusercontent.com/u/153510?v=4"},"body":"The perf framework lets the user run the test in an arbitrary repo,\nwhich makes writing a test for rebase a bit finicky.  This one uses a\nperl script to determine a longest linear segment of history, that is,\na range A..B which is completely linear.  (For a full clone of\ngit.git, this is the (whole) 'html' branch.)  The number of commits\nthat are rebased is capped at 50, however.  That is, if A..B is more\nthan 50 commits, it uses B~50..B instead.\n\nHaving determined a suitable range, it then runs rebase with all\ncombinations of rerere (on/off), -i / noninteractive, and -m / normal.\n\nSigned-off-by: Thomas Rast <trast@student.ethz.ch>\n---\n t/perf/p3400-rebase.sh |   91 ++++++++++++++++++++++++++++++++++++++++++++++++\n 1 file changed, 91 insertions(+)\n create mode 100755 t/perf/p3400-rebase.sh\n\ndiff --git a/t/perf/p3400-rebase.sh b/t/perf/p3400-rebase.sh\nnew file mode 100755\nindex 0000000..7155574\n--- /dev/null\n+++ b/t/perf/p3400-rebase.sh\n@@ -0,0 +1,91 @@\n+#!/bin/sh\n+\n+test_description=\"Tests git-rebase performance\"\n+\n+. ./perf-lib.sh\n+\n+test_perf_default_repo\n+\n+test_expect_success 'find a long run of linear history' '\n+\tgit rev-list --topo-order --parents --all |\n+\tperl -e '\\''$maxL = 0; $maxcommit = undef;\n+\t\twhile (<>) {\n+\t\t\tchomp;\n+\t\t\t@parents = split;\n+\t\t\t$commit = shift @parents;\n+\t\t\tif ($L{$commit} > $maxL) {\n+\t\t\t\t$maxL = $L{$commit};\n+\t\t\t\t$maxcommit = $commit;\n+\t\t\t}\n+\t\t\tif (1 == scalar @parents\n+\t\t\t\t&& $L{$commit} >= $L{$parents[0]}) {\n+\t\t\t\t$L{$parents[0]} = $L{$commit}+1;\n+\t\t\t\t$C{$parents[0]} = $commit;\n+\t\t\t}\n+\t\t}\n+\t\t$cur = $maxcommit;\n+\t\twhile (defined $C{$cur}) {\n+\t\t\t$cur = $C{$cur};\n+\t\t}\n+\t\tif ($maxL > 50) {\n+\t\t\t$maxL = 50;\n+\t\t}\n+\t\tprint \"$cur~$maxL\\n$cur\\n\";\n+\t'\\'' >rebase-args &&\n+\tbase_rev=$(sed -n 1p rebase-args) &&\n+\ttip_rev=$(sed -n 2p rebase-args) &&\n+\tgit checkout -f -b work $tip_rev\n+'\n+\n+export base_rev tip_rev\n+\n+test_expect_success 'verify linearity' '\n+\tgit rev-list --parents $base_rev.. >list &&\n+\t! grep \"[0-9a-f]+ [0-9a-f]+ [0-9a-f+].*\" list\n+'\n+\n+test_expect_success 'disable rerere' '\n+\tgit config rerere.enabled false\n+'\n+\n+test_perf 'rebase -f (rerere off)' '\n+\tgit rebase -f $base_rev\n+'\n+\n+test_perf 'rebase -m -f (rerere off)' '\n+\tgit rebase -m -f $base_rev\n+'\n+\n+test_perf 'rebase -i -f (rerere off)' '\n+\tGIT_EDITOR=: git rebase -i -f $base_rev\n+'\n+\n+test_perf 'rebase -i -m -f (rerere off)' '\n+\tGIT_EDITOR=: git rebase -i -m -f $base_rev\n+'\n+\n+test_expect_success 'enable rerere and prime it' '\n+\tgit config rerere.enabled true &&\n+\tgit rebase -f $base_rev &&\n+\tgit rebase -m -f $base_rev &&\n+\tGIT_EDITOR=: git rebase -i -f $base_rev &&\n+\tGIT_EDITOR=: git rebase -i -m -f $base_rev\n+'\n+\n+test_perf 'rebase -f (rerere ON)' '\n+\tgit rebase -f $base_rev\n+'\n+\n+test_perf 'rebase -m -f (rerere ON)' '\n+\tgit rebase -m -f $base_rev\n+'\n+\n+test_perf 'rebase -i -f (rerere ON)' '\n+\tGIT_EDITOR=: git rebase -i -f $base_rev\n+'\n+\n+test_perf 'rebase -i -m -f (rerere ON)' '\n+\tGIT_EDITOR=: git rebase -i -m -f $base_rev\n+'\n+\n+test_done\n-- \n1.7.10.rc0.230.g16d90\n"},{"id":"186720","messageId":"64fe6409a201285227862feffb2d51e4d8efc3eb.1331561353.git.trast@student.ethz.ch","threadId":"29918","inReplyTo":"cover.1331561353.git.trast@student.ethz.ch","subject":"[PATCH 03/11] Introduce a performance test for git-blame","fromName":"Thomas Rast","fromEmail":"trast@student.ethz.ch","sentAt":"2012-03-12T15:09:59Z","receivedAt":"2012-03-12T15:09:59Z","isPatch":true,"sender":{"key":"tr@thomasrast.ch","avatar":"https://avatars.githubusercontent.com/u/153510?v=4"},"body":"It blames a pseudo-randomly chosen (but fixed as long as the tested\nrepository remains the same) set of 10 files with various options.\nThe choice of 10 is arbitrary, but gives a nice runtime with git.git.\n\nSigned-off-by: Thomas Rast <trast@student.ethz.ch>\n---\n t/perf/p8002-blame.sh |   46 ++++++++++++++++++++++++++++++++++++++++++++++\n 1 file changed, 46 insertions(+)\n create mode 100755 t/perf/p8002-blame.sh\n\ndiff --git a/t/perf/p8002-blame.sh b/t/perf/p8002-blame.sh\nnew file mode 100755\nindex 0000000..000b9d2\n--- /dev/null\n+++ b/t/perf/p8002-blame.sh\n@@ -0,0 +1,46 @@\n+#!/bin/sh\n+\n+test_description=\"Tests blame performance\"\n+\n+. ./perf-lib.sh\n+\n+test_perf_default_repo\n+\n+# Pick 10 files to blame pseudo-randomly.  The sort key is the blob\n+# hash, so it is stable.\n+test_expect_success 'setup' '\n+\tgit ls-tree HEAD | grep ^100644 |\n+\tsort -k 3 | head | cut -f 2 >filelist\n+'\n+\n+test_perf 'blame' '\n+\twhile read -r name; do\n+\t\tgit blame HEAD -- \"$name\" >/dev/null\n+\tdone <filelist\n+'\n+\n+test_perf 'blame -M' '\n+\twhile read -r name; do\n+\t\tgit blame -M HEAD -- \"$name\" >/dev/null\n+\tdone <filelist\n+'\n+\n+test_perf 'blame -C' '\n+\twhile read -r name; do\n+\t\tgit blame -C HEAD -- \"$name\" >/dev/null\n+\tdone <filelist\n+'\n+\n+test_perf 'blame -C -C' '\n+\twhile read -r name; do\n+\t\tgit blame -C -C HEAD -- \"$name\" >/dev/null\n+\tdone <filelist\n+'\n+\n+test_perf 'blame -C -C -C' '\n+\twhile read -r name; do\n+\t\tgit blame -C -C -C HEAD -- \"$name\" >/dev/null\n+\tdone <filelist\n+'\n+\n+test_done\n-- \n1.7.10.rc0.230.g16d90\n"},{"id":"186717","messageId":"ac09e8e93b110382409a8cd9d6c29189056da100.1331561353.git.trast@student.ethz.ch","threadId":"29918","inReplyTo":"cover.1331561353.git.trast@student.ethz.ch","subject":"[PATCH 04/11] perf: display average instead of minimum time","fromName":"Thomas Rast","fromEmail":"trast@student.ethz.ch","sentAt":"2012-03-12T15:10:00Z","receivedAt":"2012-03-12T15:10:00Z","isPatch":true,"sender":{"key":"tr@thomasrast.ch","avatar":"https://avatars.githubusercontent.com/u/153510?v=4"},"body":"The perf tests initially only saved the minimum measurement (chosen\naccording to wall time).  This is common for timings, as it tends to\nmeasure the shortest time in which the machine can run the test.  It\nis also a bit more forgiving for the user if he ran some other task\nduring one of the tests.\n\nHowever, experiments with p3400-rebase.sh showed that for\nlonger-running tasks with a lot of kernel involvement (such as a shell\nscript and its constant forking) the variance is so high that the\nminimum becomes extremely unstable.  The best-of-10 (!) timings\nfluctuated by around 5% in extreme cases.\n\nSwitch to the average, as that tends to cancel out the variance for\nsufficiently large numbers of $GIT_PERF_REPEAT_COUNT.  The catch is\nthat now we can no longer write off the initial (possibly) cache-cold\ntiming.  So the test is now run 1+n times, and the first run is\ndiscarded.\n\nSince we're already rewriting that paragraph, also correctly state the\ndefault value of GIT_PERF_REPEAT_COUNT, which is 3.\n\nThe shift of the averaging logic to aggregate.perl is not, strictly\nspeaking, necessary.  However, min_time.perl would have had to be\nrenamed anyway.  It also sets the stage for the next patch.\n\nSigned-off-by: Thomas Rast <trast@student.ethz.ch>\n---\n t/perf/README         |    8 +++++---\n t/perf/aggregate.perl |   25 +++++++++++++++++--------\n t/perf/min_time.perl  |   21 ---------------------\n t/perf/perf-lib.sh    |    5 +++--\n 4 files changed, 25 insertions(+), 34 deletions(-)\n delete mode 100755 t/perf/min_time.perl\n\ndiff --git a/t/perf/README b/t/perf/README\nindex b2dbad4..fa94eb5 100644\n--- a/t/perf/README\n+++ b/t/perf/README\n@@ -54,9 +54,11 @@ anything beforehand.\n \n You can set the following variables (also in your config.mak):\n \n-    GIT_PERF_REPEAT_COUNT\n-\tNumber of times a test should be repeated for best-of-N\n-\tmeasurements.  Defaults to 5.\n+    GIT_PERF_REPEAT_COUNT='n'\n+\tNumber of times a test should be repeated.  The test is run\n+\t'n+1' times: once to warm up the caches and 'n' more times to\n+\tgather the measurements.  The first run is discarded, and the\n+\tother 'n' runs are averaged.  Defaults to 3.\n \n     GIT_PERF_MAKE_OPTS\n \tOptions to use when automatically building a git tree for\ndiff --git a/t/perf/aggregate.perl b/t/perf/aggregate.perl\nindex c0afa0b..9c781fa 100755\n--- a/t/perf/aggregate.perl\n+++ b/t/perf/aggregate.perl\n@@ -10,13 +10,22 @@\n sub get_times {\n \tmy $name = shift;\n \topen my $fh, \"<\", $name or return undef;\n-\tmy $line = <$fh>;\n-\treturn undef if not defined $line;\n+\tmy $sum_rt = 0.0;\n+\tmy $sum_u = 0.0;\n+\tmy $sum_s = 0.0;\n+\tmy $n = 0;\n+\twhile (<$fh>) {\n+\t\t/^(?:(\\d+):)?(\\d+):(\\d+(?:\\.\\d+)?) (\\d+(?:\\.\\d+)?) (\\d+(?:\\.\\d+)?)$/\n+\t\t\tor die \"bad input line: $_\";\n+\t\tmy $rt = ((defined $1 ? $1 : 0.0)*60+$2)*60+$3;\n+\t\t$sum_rt += $rt;\n+\t\t$sum_u += $4;\n+\t\t$sum_s += $5;\n+\t\t$n++;\n+\t}\n+\treturn undef if !$n;\n \tclose $fh or die \"cannot close $name: $!\";\n-\t$line =~ /^(?:(\\d+):)?(\\d+):(\\d+(?:\\.\\d+)?) (\\d+(?:\\.\\d+)?) (\\d+(?:\\.\\d+)?)$/\n-\t\tor die \"bad input line: $line\";\n-\tmy $rt = ((defined $1 ? $1 : 0.0)*60+$2)*60+$3;\n-\treturn ($rt, $4, $5);\n+\treturn ($sum_rt/$n, $sum_u/$n, $sum_s/$n);\n }\n \n sub format_times {\n@@ -140,8 +149,8 @@ sub have_slash {\n \tfor my $i (0..$#dirs) {\n \t\tmy $d = $dirs[$i];\n \t\t$times{$prefixes{$d}.$t} = [get_times(\"test-results/$prefixes{$d}$t.times\")];\n-\t\tmy ($r,$u,$s) = @{$times{$prefixes{$d}.$t}};\n-\t\tmy $w = length format_times($r,$u,$s,$firstr);\n+\t\tmy ($r,$u,$s,$sign) = @{$times{$prefixes{$d}.$t}};\n+\t\tmy $w = length format_times($r,$u,$s,$sign,$firstr);\n \t\t$colwidth[$i] = $w if $w > $colwidth[$i];\n \t\t$firstr = $r unless defined $firstr;\n \t}\ndiff --git a/t/perf/min_time.perl b/t/perf/min_time.perl\ndeleted file mode 100755\nindex c1a2717..0000000\n--- a/t/perf/min_time.perl\n+++ /dev/null\n@@ -1,21 +0,0 @@\n-#!/usr/bin/perl\n-\n-my $minrt = 1e100;\n-my $min;\n-\n-while (<>) {\n-\t# [h:]m:s.xx U.xx S.xx\n-\t/^(?:(\\d+):)?(\\d+):(\\d+(?:\\.\\d+)?) (\\d+(?:\\.\\d+)?) (\\d+(?:\\.\\d+)?)$/\n-\t\tor die \"bad input line: $_\";\n-\tmy $rt = ((defined $1 ? $1 : 0.0)*60+$2)*60+$3;\n-\tif ($rt < $minrt) {\n-\t\t$min = $_;\n-\t\t$minrt = $rt;\n-\t}\n-}\n-\n-if (!defined $min) {\n-\tdie \"no input found\";\n-}\n-\n-print $min;\ndiff --git a/t/perf/perf-lib.sh b/t/perf/perf-lib.sh\nindex 5580c22..a13f105 100644\n--- a/t/perf/perf-lib.sh\n+++ b/t/perf/perf-lib.sh\n@@ -163,7 +163,7 @@ test_perf () {\n \t\telse\n \t\t\techo \"perf $test_count - $1:\"\n \t\tfi\n-\t\tfor i in $(seq 1 $GIT_PERF_REPEAT_COUNT); do\n+\t\tfor i in $(seq 0 $GIT_PERF_REPEAT_COUNT); do\n \t\t\tsay >&3 \"running: $2\"\n \t\t\tif test_run_perf_ \"$2\"\n \t\t\tthen\n@@ -184,7 +184,8 @@ test_perf () {\n \t\t\ttest_ok_ \"$1\"\n \t\tfi\n \t\tbase=\"$perf_results_dir\"/\"$perf_results_prefix$(basename \"$0\" .sh)\".\"$test_count\"\n-\t\t\"$TEST_DIRECTORY\"/perf/min_time.perl test_time.* >\"$base\".times\n+\t\trm test_time.0\n+\t\tcat test_time.* >\"$base\".times\n \tfi\n \techo >&3 \"\"\n }\n-- \n1.7.10.rc0.230.g16d90\n"},{"id":"186713","messageId":"f6d2a82ece115f05d2fc7ff8214c39cf530373c0.1331561353.git.trast@student.ethz.ch","threadId":"29918","inReplyTo":"cover.1331561353.git.trast@student.ethz.ch","subject":"[PATCH 05/11] perf: suppress aggregation also in 'run'","fromName":"Thomas Rast","fromEmail":"trast@student.ethz.ch","sentAt":"2012-03-12T15:10:01Z","receivedAt":"2012-03-12T15:10:01Z","isPatch":true,"sender":{"key":"tr@thomasrast.ch","avatar":"https://avatars.githubusercontent.com/u/153510?v=4"},"body":"perf-lib.sh avoids calling aggregate.perl on the results if\n$GIT_PERF_AGGREGATING_LATER is set.  'run' uses this to suppress the\ntable for test runs, so that it can display them all together at the\nend.\n\nHowever, other users may want to do the same while still benefiting\nfrom run's facilities such as automatic compilation/testing of\narbitrary revisions.  So teach 'run' the same interface: when\nGIT_PERF_AGGREGATING_LATER is set, do not call aggregate.perl.\n\nSigned-off-by: Thomas Rast <trast@student.ethz.ch>\n---\n t/perf/run |    3 ++-\n 1 file changed, 2 insertions(+), 1 deletion(-)\n\ndiff --git a/t/perf/run b/t/perf/run\nindex cfd7012..886290b 100755\n--- a/t/perf/run\n+++ b/t/perf/run\n@@ -69,6 +69,7 @@ run_dirs () {\n \tdone\n }\n \n+orig_GIT_AGGREGATING_LATER=$GIT_PERF_AGGREGATING_LATER\n GIT_PERF_AGGREGATING_LATER=t\n export GIT_PERF_AGGREGATING_LATER\n \n@@ -79,4 +80,4 @@ if test $# = 0 -o \"$1\" = -- -o -f \"$1\"; then\n \tset -- . \"$@\"\n fi\n run_dirs \"$@\"\n-./aggregate.perl \"$@\"\n+test -z \"$orig_GIT_AGGREGATING_LATER\" && ./aggregate.perl \"$@\"\n-- \n1.7.10.rc0.230.g16d90\n"},{"id":"186714","messageId":"2d1921e1faf6e5927edd6c620ba71b07e8965979.1331561353.git.trast@student.ethz.ch","threadId":"29918","inReplyTo":"cover.1331561353.git.trast@student.ethz.ch","subject":"[PATCH 06/11] perf: dereference to a commit when building","fromName":"Thomas Rast","fromEmail":"trast@student.ethz.ch","sentAt":"2012-03-12T15:10:02Z","receivedAt":"2012-03-12T15:10:02Z","isPatch":true,"sender":{"key":"tr@thomasrast.ch","avatar":"https://avatars.githubusercontent.com/u/153510?v=4"},"body":"So far the automatic compilation in 'run' just ran rev-parse on the\nuser input.  This means the user could test once for v1.7.9 and once\nfor 828ea97 (Git 1.7.9, 2012-01-27) and the script would never realize\nthat they are actually the same.  Even more confusingly, the user\ncould test 828ea97d and ask about './aggregate.perl v1.7.9' and it\nwould not find any results.\n\nDereference to a commit when constructing the directory name to avoid\nthis.\n\nSigned-off-by: Thomas Rast <trast@student.ethz.ch>\n---\n t/perf/aggregate.perl |    2 +-\n t/perf/run            |    2 +-\n 2 files changed, 2 insertions(+), 2 deletions(-)\n\ndiff --git a/t/perf/aggregate.perl b/t/perf/aggregate.perl\nindex 9c781fa..4586840 100755\n--- a/t/perf/aggregate.perl\n+++ b/t/perf/aggregate.perl\n@@ -52,7 +52,7 @@ sub format_times {\n \tmy $dir;\n \tlast if -f $arg or $arg eq \"--\";\n \tif (! -d $arg) {\n-\t\tmy $rev = Git::command_oneline(qw(rev-parse --verify), $arg);\n+\t\tmy $rev = Git::command_oneline(qw(rev-parse --verify), $arg.\"^{commit}\");\n \t\t$dir = \"build/\".$rev;\n \t} else {\n \t\t$arg =~ s{/*$}{};\ndiff --git a/t/perf/run b/t/perf/run\nindex 886290b..e4f9c22 100755\n--- a/t/perf/run\n+++ b/t/perf/run\n@@ -45,7 +45,7 @@ run_dirs_helper () {\n \t\tshift\n \tfi\n \tif [ ! -d \"$mydir\" ]; then\n-\t\trev=$(git rev-parse --verify \"$mydir\" 2>/dev/null) ||\n+\t\trev=$(git rev-parse --verify \"$mydir^{commit}\" 2>/dev/null) ||\n \t\tdie \"'$mydir' is neither a directory nor a valid revision\"\n \t\tif [ ! -d build/$rev ]; then\n \t\t\tunpack_git_rev $rev\n-- \n1.7.10.rc0.230.g16d90\n"},{"id":"186718","messageId":"f9c799d3e926c1ec650ee7d414ba4d1040082f36.1331561353.git.trast@student.ethz.ch","threadId":"29918","inReplyTo":"cover.1331561353.git.trast@student.ethz.ch","subject":"[PATCH 07/11] perf: convert realtime to seconds when collecting runs","fromName":"Thomas Rast","fromEmail":"trast@student.ethz.ch","sentAt":"2012-03-12T15:10:03Z","receivedAt":"2012-03-12T15:10:03Z","isPatch":true,"sender":{"key":"tr@thomasrast.ch","avatar":"https://avatars.githubusercontent.com/u/153510?v=4"},"body":"Our use of 'time' gives results along the lines of\n\n  0:01.04 0.67 0.02\n\nThat is, the realtime field is formatted as a H:M:S.CC time with the\nhour field optional.  Use a little perl magic to convert it to\nseconds, too.  This makes it easier to parse further downstream.\n\nSigned-off-by: Thomas Rast <trast@student.ethz.ch>\n---\n t/perf/aggregate.perl |    9 ++++-----\n t/perf/perf-lib.sh    |    3 ++-\n 2 files changed, 6 insertions(+), 6 deletions(-)\n\ndiff --git a/t/perf/aggregate.perl b/t/perf/aggregate.perl\nindex 4586840..b04d3a0 100755\n--- a/t/perf/aggregate.perl\n+++ b/t/perf/aggregate.perl\n@@ -15,12 +15,11 @@ sub get_times {\n \tmy $sum_s = 0.0;\n \tmy $n = 0;\n \twhile (<$fh>) {\n-\t\t/^(?:(\\d+):)?(\\d+):(\\d+(?:\\.\\d+)?) (\\d+(?:\\.\\d+)?) (\\d+(?:\\.\\d+)?)$/\n+\t\t/^(\\d+(?:\\.\\d+)?) (\\d+(?:\\.\\d+)?) (\\d+(?:\\.\\d+)?)$/\n \t\t\tor die \"bad input line: $_\";\n-\t\tmy $rt = ((defined $1 ? $1 : 0.0)*60+$2)*60+$3;\n-\t\t$sum_rt += $rt;\n-\t\t$sum_u += $4;\n-\t\t$sum_s += $5;\n+\t\t$sum_rt += $1;\n+\t\t$sum_u += $2;\n+\t\t$sum_s += $3;\n \t\t$n++;\n \t}\n \treturn undef if !$n;\ndiff --git a/t/perf/perf-lib.sh b/t/perf/perf-lib.sh\nindex a13f105..bfc2926 100644\n--- a/t/perf/perf-lib.sh\n+++ b/t/perf/perf-lib.sh\n@@ -185,7 +185,8 @@ test_perf () {\n \t\tfi\n \t\tbase=\"$perf_results_dir\"/\"$perf_results_prefix$(basename \"$0\" .sh)\".\"$test_count\"\n \t\trm test_time.0\n-\t\tcat test_time.* >\"$base\".times\n+\t\tperl -pe 's/(?:(\\d+):)?(\\d+):(\\d+(?:\\.\\d+)?)/((defined $1?$1:0)*60+$2)*60+$3/e' \\\n+\t\t\ttest_time.* >\"$base\".times\n \tfi\n \techo >&3 \"\"\n }\n-- \n1.7.10.rc0.230.g16d90\n"},{"id":"186715","messageId":"8efb03c8da7e76a1fd39a3bdcfd37ea0dc9bda72.1331561353.git.trast@student.ethz.ch","threadId":"29918","inReplyTo":"cover.1331561353.git.trast@student.ethz.ch","subject":"[PATCH 08/11] perf/aggregate: optionally include a t-test score","fromName":"Thomas Rast","fromEmail":"trast@student.ethz.ch","sentAt":"2012-03-12T15:10:04Z","receivedAt":"2012-03-12T15:10:04Z","isPatch":true,"sender":{"key":"tr@thomasrast.ch","avatar":"https://avatars.githubusercontent.com/u/153510?v=4"},"body":"Adds support for calling out to R[1] to perform a significance test on\nthe gathered results.\n\nMore specifically, the script's purpose was always to compare several\nrevisions (it still makes nice tables for single tests though ;-).\nGiven the first and some other column (= revision/directory), this\nruns Welch's t-test[2] on the two sets of measurements to determine\nwhether there is a significant difference between the distributions.\nIt then shows the p-value in a simplified form, so that significant\ndifferences stand out optically.\n\nAll of this is entirely optional: if R is not available, it simply\nputs nothing in this field.\n\n[1] http://www.r-project.org/\n[2] http://en.wikipedia.org/wiki/Welch%27s_t-test\n\nSigned-off-by: Thomas Rast <trast@student.ethz.ch>\n---\n t/perf/aggregate.perl  |   37 +++++++++++++++++++++++++++++++------\n t/perf/t_test_score.sh |   24 ++++++++++++++++++++++++\n 2 files changed, 55 insertions(+), 6 deletions(-)\n create mode 100755 t/perf/t_test_score.sh\n\ndiff --git a/t/perf/aggregate.perl b/t/perf/aggregate.perl\nindex b04d3a0..4db685d 100755\n--- a/t/perf/aggregate.perl\n+++ b/t/perf/aggregate.perl\n@@ -7,8 +7,12 @@\n \t\"$FindBin::Bin/../../perl/blib/arch/auto/Git\";\n use Git;\n \n+my $any_sign_printed = 0;\n+\n sub get_times {\n \tmy $name = shift;\n+\tmy $firstset = shift;\n+\tmy $sig = \"\";\n \topen my $fh, \"<\", $name or return undef;\n \tmy $sum_rt = 0.0;\n \tmy $sum_u = 0.0;\n@@ -24,11 +28,21 @@ sub get_times {\n \t}\n \treturn undef if !$n;\n \tclose $fh or die \"cannot close $name: $!\";\n-\treturn ($sum_rt/$n, $sum_u/$n, $sum_s/$n);\n+\tif (defined $firstset &&\n+\t    open my $ph, \"-|\", \"./t_test_score.sh $name $firstset 2>/dev/null\") {\n+\t\tmy $result = <$ph>;\n+\t\tclose $ph or die \"cannot close pipe to t_test_score.sh: $!\";\n+\t\tchomp $result;\n+\t\t$sig = $result;\n+\t\tif ($sig ne \"\") {\n+\t\t\t$any_sign_printed = 1;\n+\t\t}\n+\t}\n+\treturn ($sum_rt/$n, $sum_u/$n, $sum_s/$n, $sig);\n }\n \n sub format_times {\n-\tmy ($r, $u, $s, $firstr) = @_;\n+\tmy ($r, $u, $s, $sign, $firstr) = @_;\n \tif (!defined $r) {\n \t\treturn \"<missing>\";\n \t}\n@@ -41,6 +55,7 @@ sub format_times {\n \t\t} else {\n \t\t\t$out .= \" +inf\";\n \t\t}\n+\t\t$out .= $sign;\n \t}\n \treturn $out;\n }\n@@ -145,13 +160,17 @@ sub have_slash {\n }\n for my $t (@subtests) {\n \tmy $firstr;\n+\tmy $firstset;\n \tfor my $i (0..$#dirs) {\n \t\tmy $d = $dirs[$i];\n-\t\t$times{$prefixes{$d}.$t} = [get_times(\"test-results/$prefixes{$d}$t.times\")];\n+\t\t$times{$prefixes{$d}.$t} = [get_times(\"test-results/$prefixes{$d}$t.times\", $firstset)];\n \t\tmy ($r,$u,$s,$sign) = @{$times{$prefixes{$d}.$t}};\n \t\tmy $w = length format_times($r,$u,$s,$sign,$firstr);\n \t\t$colwidth[$i] = $w if $w > $colwidth[$i];\n-\t\t$firstr = $r unless defined $firstr;\n+\t\tif (!defined $firstr) {\n+\t\t\t$firstr = $r;\n+\t\t\t$firstset = \"test-results/$prefixes{$d}$t.times\";\n+\t\t}\n \t}\n }\n my $totalwidth = 3*@dirs+$descrlen;\n@@ -169,9 +188,15 @@ sub have_slash {\n \tmy $firstr;\n \tfor my $i (0..$#dirs) {\n \t\tmy $d = $dirs[$i];\n-\t\tmy ($r,$u,$s) = @{$times{$prefixes{$d}.$t}};\n-\t\tprintf \"   %-$colwidth[$i]s\", format_times($r,$u,$s,$firstr);\n+\t\tmy ($r,$u,$s,$sign) = @{$times{$prefixes{$d}.$t}};\n+\t\tprintf \"   %-$colwidth[$i]s\", format_times($r,$u,$s,$sign,$firstr);\n \t\t$firstr = $r unless defined $firstr;\n \t}\n \tprint \"\\n\";\n }\n+\n+if ($any_sign_printed) {\n+\tprint \"-\"x$totalwidth, \"\\n\";\n+\tprint \"Significance hints:  '.' 0.1  '*' 0.05  '**' 0.01  '***' 0.001\\n\"\n+}\n+\ndiff --git a/t/perf/t_test_score.sh b/t/perf/t_test_score.sh\nnew file mode 100755\nindex 0000000..32353d6\n--- /dev/null\n+++ b/t/perf/t_test_score.sh\n@@ -0,0 +1,24 @@\n+#!/bin/sh\n+\n+# If the user doesn't have R, we don't care\n+\n+command -v R >/dev/null || exit 0\n+\n+# Uses R to run a t-test on the hypothesis that the elapsed time\n+# values in $1 are less than the ones in $2.\n+\n+pvalue=$(R --no-save --slave <<-EOF\n+\ta <- read.table(\"$1\")\n+\tb <- read.table(\"$2\")\n+\ttst <- t.test(a\\$V1, b\\$V1)\n+\tp <- tst\\$p.value\n+\tif (p<0.001) print (\"***\") \\\n+\telse if (p<0.01) print (\"**\") \\\n+\telse if (p<0.05) print (\"*\") \\\n+\telse if (p<0.1) print (\".\")\n+EOF\n+)\n+\n+pvalue=${pvalue##\\[1\\] \\\"}\n+pvalue=${pvalue%%\\\"}\n+echo \"$pvalue\"\n-- \n1.7.10.rc0.230.g16d90\n"},{"id":"186712","messageId":"20160e2ec785353a5e1203a7c769b0bc5881b83d.1331561353.git.trast@student.ethz.ch","threadId":"29918","inReplyTo":"cover.1331561353.git.trast@student.ethz.ch","subject":"[PATCH 09/11] perf/run: allow skipping some revisions","fromName":"Thomas Rast","fromEmail":"trast@student.ethz.ch","sentAt":"2012-03-12T15:10:05Z","receivedAt":"2012-03-12T15:10:05Z","isPatch":true,"sender":{"key":"tr@thomasrast.ch","avatar":"https://avatars.githubusercontent.com/u/153510?v=4"},"body":"You may want to compare two revisions/dirs A and B, but have already\ntested one of them, say A.  './run A B' would re-run the tests for\nboth, which is very time-consuming.  You could invoke\n\n  ./run B && ./aggregate.perl A B\n\nbut that is tedious.\n\nMake it so that a ^ prefix means \"aggregate this revision, but do not\ntest it\"; i.e., you can say\n\n  ./run ^A B\n\nto achieve what you want.\n\nSigned-off-by: Thomas Rast <trast@student.ethz.ch>\n---\n t/perf/aggregate.perl |    1 +\n t/perf/run            |    5 +++++\n 2 files changed, 6 insertions(+)\n\ndiff --git a/t/perf/aggregate.perl b/t/perf/aggregate.perl\nindex 4db685d..747f885 100755\n--- a/t/perf/aggregate.perl\n+++ b/t/perf/aggregate.perl\n@@ -65,6 +65,7 @@ sub format_times {\n \tmy $arg = $ARGV[0];\n \tmy $dir;\n \tlast if -f $arg or $arg eq \"--\";\n+\t$arg =~ s/^\\^// if (! -d $arg);\n \tif (! -d $arg) {\n \t\tmy $rev = Git::command_oneline(qw(rev-parse --verify), $arg.\"^{commit}\");\n \t\t$dir = \"build/\".$rev;\ndiff --git a/t/perf/run b/t/perf/run\nindex e4f9c22..bc66067 100755\n--- a/t/perf/run\n+++ b/t/perf/run\n@@ -37,6 +37,11 @@ build_git_rev () {\n \n run_dirs_helper () {\n \tmydir=${1%/}\n+\tcase \"$mydir\" in\n+\t\t^*)\n+\t\t\treturn\n+\t\t\t;;\n+\tesac\n \tshift\n \twhile test $# -gt 0 -a \"$1\" != -- -a ! -f \"$1\"; do\n \t\tshift\n-- \n1.7.10.rc0.230.g16d90\n"},{"id":"186726","messageId":"CACsJy8BQXvX6iOH0txcw5wmW-2ddnG3A39BxsrspfkqPTwDy_g@mail.gmail.com","threadId":"29918","inReplyTo":"cover.1331561353.git.trast@student.ethz.ch","subject":"Re: [PATCH 00/11] perf improvements past v1.7.10","fromName":"Nguyen Thai Ngoc Duy","fromEmail":"pclouds@gmail.com","sentAt":"2012-03-12T15:28:45Z","receivedAt":"2012-03-12T15:28:45Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"Not really related to the series, how do I compare the same tests on\ndifferent repositories? These repositories share the same DAG, but may\nhave different object representation (all loose, single pack, zillion\nof packs, different versions of pack...)\n-- \nDuy\n"},{"id":"186727","messageId":"m3d38hztqt.fsf@localhost.localdomain","threadId":"29918","inReplyTo":"cover.1331561353.git.trast@student.ethz.ch","subject":"Re: [PATCH 00/11] perf improvements past v1.7.10","fromName":"Jakub Narebski","fromEmail":"jnareb@gmail.com","sentAt":"2012-03-12T16:09:19Z","receivedAt":"2012-03-12T16:09:19Z","isPatch":true,"sender":{"key":"jnareb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/2706?v=4"},"body":"Thomas Rast <trast@student.ethz.ch> writes:\n\n> This is what I have collected over a week of playing with the perf\n> suite.  There is no rush; I am just sending them out for anyone\n> interested.  If you plan on using the perf suite, I would still\n> suggest you apply them.\n> \n> Aside from the new tests, the main goal is the bisection script.\n> [8/11] is fun too.  Try it!  You will have to install GNU R however.\n[...]\n\n>   perf/aggregate: optionally include a t-test score\n\nBy the way, have you heard about Dumbbench module on CPAN?  It is a\nmodule that attempts to implement reasonably robust benchmarking,\ntrying to find and discard bad runs (outliers):\n\n  http://search.cpan.org/perldoc?Dumbbench\n\nIt might be an easier solution (thanks to wonders of local::lib) than\nGNU R statistical package.\n\n[...]\n>   perf: implement a test-selection feature\n>   perf: add a bisection tool\n\nIt looks like those last two (10/11, 11/11) didn't made it to git\nmailing list, probably because of anti-spam size limits of email\nmessage on vger.\n\nDo you have a public repository with this branch somewhere?\n\n-- \nJakub Narebski\n"},{"id":"186731","messageId":"b21f1ec0d40a38be53040d182933ed2935bbcfc9.1331561353.git.trast@student.ethz.ch","threadId":"29918","inReplyTo":"cover.1331561353.git.trast@student.ethz.ch","subject":"[PATCH 10/11] perf: implement a test-selection feature","fromName":"Thomas Rast","fromEmail":"trast@student.ethz.ch","sentAt":"2012-03-12T16:27:28Z","receivedAt":"2012-03-12T16:27:28Z","isPatch":true,"sender":{"key":"tr@thomasrast.ch","avatar":"https://avatars.githubusercontent.com/u/153510?v=4"},"body":"When bisecting for a performance regression, there is usually no point\nin running all tests in the pNNNN-foo.sh file, since you are\ninterested only in the result of a single test.\n\nMake GIT_PERF_TEST_ONLY=pNNNN.I select only the test NNNN.I for\nrunning.  All other test_perf blocks are skipped.  test_expect_success\nblocks are deliberately not affected; presumably they consist of setup\ncode.\n\nSigned-off-by: Thomas Rast <trast@student.ethz.ch>\n---\n t/perf/perf-lib.sh |    6 +++++-\n 1 file changed, 5 insertions(+), 1 deletion(-)\n\ndiff --git a/t/perf/perf-lib.sh b/t/perf/perf-lib.sh\nindex bfc2926..ddb3272 100644\n--- a/t/perf/perf-lib.sh\n+++ b/t/perf/perf-lib.sh\n@@ -153,7 +153,11 @@ test_perf () {\n \ttest \"$#\" = 2 ||\n \terror \"bug in the test script: not 2 or 3 parameters to test-expect-success\"\n \texport test_prereq\n-\tif ! test_skip \"$@\"\n+\tskipping=\n+\ttest_skip \"$@\" && skipping=t\n+\ttest -z \"$GIT_PERF_TEST_ONLY\" ||\n+\ttest \"$GIT_PERF_TEST_ONLY\" = $this_test.$test_count || skipping=t\n+\tif test -z \"$skipping\"\n \tthen\n \t\tbase=$(basename \"$0\" .sh)\n \t\techo \"$test_count\" >>\"$perf_results_dir\"/$base.subtests\n-- \n1.7.10.rc0.230.g16d90\n"},{"id":"186730","messageId":"4d8e9b13f6fe755ed16c3f8f180a8f4cf4ec7199.1331561353.git.trast@student.ethz.ch","threadId":"29918","inReplyTo":"cover.1331561353.git.trast@student.ethz.ch","subject":"[PATCH 11/11] perf: add a bisection tool","fromName":"Thomas Rast","fromEmail":"trast@student.ethz.ch","sentAt":"2012-03-12T16:27:29Z","receivedAt":"2012-03-12T16:27:29Z","isPatch":true,"sender":{"key":"tr@thomasrast.ch","avatar":"https://avatars.githubusercontent.com/u/153510?v=4"},"body":"Using the perf suite for bisection is not hard with all the features\nrecently added; you can say\n\n  git bisect start --no-checkout\n  git bisect bad BAD\n  git bisect good GOOD\n\n  # repeat until done:\n  ./run BISECT_HEAD ^BAD ^GOOD\n  git bisect {good|bad}\n\nThis script does the bulk of the work for the user.  You can for\nexample say\n\n  ./bisect_slowdown p4000.3 v1.7.0 v1.7.10-rc0\n\nto bisect the ~5% performance hit incurred by 3e5a188 (diff.c: Ensure\n\"index $from..$to\" line contains unambiguous SHA1s, 2010-05-30).  The\nmain output is quite noisy, but you can follow test-results/bisect.log\nfor a more concise summary of the results and decisions taken.\n\nThe heuristic applied is that if the result for BISECT_HEAD is scaled\nbetween GOOD and BAD, then anything <0.25 must be good and >0.75 must\nbe bad.  Put in terms of differences towards the good and bad sides,\nit decides in favor of the smaller one if they are at least a factor\nof 3 apart.\n\nAnything in the middle punts and asks for help.  You may then for\nexample re-run with more iterations to improve the measurements, or\nmanually edit the test script to make it more expensive (but remember\nthat you also need to re-test BAD and GOOD in this case!).  Once you\nhave told git-bisect your decision, you can restart the script to let\nit continue the search.\n\nPoints of note:\n\n* Performance regressions are not a binary event, so the search may\n  not find anything reasonable.  For example, there is a ~6% hit in\n  p3400.4 between adc3b2b27..a9f578685, but because it is spread\n  across 13 commits, a bisection can only determine the general area.\n\n* The default settings do not have high enough S/N in the timings to\n  give good results.  Try setting GIT_PERF_REPEAT_COUNT to around 8.\n\n* Many CPUs have a dynamic overclocking feature (such as Intel Turbo\n  Boost) that helps single-process performance.  Running other things\n  in parallel may prevent it from working.  The difference may be\n  larger than 20% and will usually drown out the performance hit you\n  are bisecting for.\n\nSigned-off-by: Thomas Rast <trast@student.ethz.ch>\n---\n t/perf/bisect_slowdown |   88 ++++++++++++++++++++++++++++++++++++++++++++++++\n 1 file changed, 88 insertions(+)\n create mode 100755 t/perf/bisect_slowdown\n\ndiff --git a/t/perf/bisect_slowdown b/t/perf/bisect_slowdown\nnew file mode 100755\nindex 0000000..8b58b8d\n--- /dev/null\n+++ b/t/perf/bisect_slowdown\n@@ -0,0 +1,88 @@\n+#!/bin/sh\n+\n+test=$1\n+good=$2\n+bad=$3\n+\n+subtest=${test#p????.}\n+maintest=${test%%.*}\n+\n+export subtest maintest\n+\n+export GIT_PERF_TEST_ONLY=$test\n+\n+goodrev=$(git rev-parse \"$good^{commit}\")\n+badrev=$(git rev-parse \"$bad^{commit}\")\n+\n+git_dir=$(git rev-parse --git-dir)\n+perfdir=$(cd \"$(dirname \"$0\")\" && pwd)\n+log=\"$perfdir/test-results/bisect.log\"\n+\n+export good bad goodrev badrev perfdir log\n+\n+if ! test -s \"$git_dir/BISECT_LOG\"; then\n+\t# Establish initial timings for the endpoints\n+\techo \"## Before bisection: comparing $good (good) and $bad (bad)\" >\"$log\"\n+\tGIT_PERF_AGGREGATING_LATER=t ./run \"$good\" \"$bad\" -- \"$maintest\"*.sh\n+\t./aggregate.perl \"$good\" \"$bad\" -- \"$maintest\"*.sh | tee -a \"$log\"\n+\n+\tcd \"$perfdir/../..\"\n+\n+\techo \"## Starting bisection between $good (good) and $bad (bad)\" >>\"$log\"\n+\tgit bisect start --no-checkout\n+\tgit bisect good \"$good\"\n+\tgit bisect bad \"$bad\"\n+else\n+\tcd \"$perfdir/../..\"\n+fi\n+\n+git bisect run sh -c '\n+\tcd \"$perfdir\"\n+\techo \"## Testing $(git log --no-walk --format=\"%h %s\" BISECT_HEAD)\" >>\"$log\"\n+\tGIT_PERF_AGGREGATING_LATER=t ./run BISECT_HEAD -- \"$maintest\"*.sh\n+\t./aggregate.perl BISECT_HEAD \"$good\" \"$bad\" -- \"$maintest\"*.sh | tee -a \"$log\"\n+\ttested=$(git rev-parse BISECT_HEAD)\n+\tperl -e '\\''\n+\t\topen STDOUT, \"|-\", \"tee\", \"-a\", \"'\"$log\"'\" or die \"failed to spawn tee: $!\";\n+\t\tsub avg {\n+\t\t\topen my $fp, \"<$_\" or die \"cannot open $_: $!\";\n+\t\t\tmy $s = 0;\n+\t\t\tmy $n = 0;\n+\t\t\twhile (<$fp>) {\n+\t\t\t\tif (/^(\\d+\\.\\d+)/) {\n+\t\t\t\t\t$s += $1;\n+\t\t\t\t\t$n++;\n+\t\t\t\t}\n+\t\t\t}\n+\t\t\tclose $fp or die \"cannot close: $!\";\n+\t\t\treturn $s/$n;\n+\t\t}\n+\t\tmy ($good,$bad,$test) = map { avg $_ } @ARGV;\n+\t\tif ($good == $bad) { die \"good = bad?\" };\n+\t\tmy $ratio = ($test-$good)/($bad-$good);\n+\t\tif ($ratio < 0.25) {\n+\t\t\tprint \"# Result: GOOD!\\n\";\n+\t\t\texit 0;\n+\t\t} elsif ($ratio > 0.75) {\n+\t\t\tprint \"# Result: BAD!\\n\";\n+\t\t\texit 42;\n+\t\t} else {\n+\t\t\tprint \"# INCONCLUSIVE.\\n\";\n+\t\t\tprint \"# Please check manually, say git bisect good/bad, and restart me!\\n\";\n+\t\t\texit 1;\n+\t\t}\n+\t'\\'' \\\n+\t\ttest-results/build_\"$goodrev\".\"$maintest\"*.\"$subtest\".times \\\n+\t\ttest-results/build_\"$badrev\".\"$maintest\"*.\"$subtest\".times \\\n+\t\ttest-results/build_\"$tested\".\"$maintest\"*.\"$subtest\".times\n+\tret=$?\n+\ttest $ret = 0 && exit 0\n+\ttest $ret = 42 && exit 1\n+\texit 255\n+' || exit $?\n+\n+cd \"$perfdir\"\n+echo \"## Double-checking $(git log --no-walk --format=\"%h %s\" bisect/bad)\" >>\"$log\"\n+GIT_PERF_AGGREGATING_LATER=t ./run bisect/bad^ bisect/bad -- \"$maintest\"*.sh\n+git show bisect/bad\n+./aggregate.perl bisect/bad^ bisect/bad -- \"$maintest\"*.sh | tee -a \"$log\"\n-- \n1.7.10.rc0.230.g16d90\n"},{"id":"186736","messageId":"87sjhd6atl.fsf@thomas.inf.ethz.ch","threadId":"29918","inReplyTo":"m3d38hztqt.fsf@localhost.localdomain","subject":"Re: [PATCH 00/11] perf improvements past v1.7.10","fromName":"Thomas Rast","fromEmail":"trast@inf.ethz.ch","sentAt":"2012-03-12T16:30:46Z","receivedAt":"2012-03-12T16:30:46Z","isPatch":true,"sender":{"key":"tr@thomasrast.ch","avatar":"https://avatars.githubusercontent.com/u/153510?v=4"},"body":"Jakub Narebski <jnareb@gmail.com> writes:\n\n> Thomas Rast <trast@student.ethz.ch> writes:\n>\n>>   perf/aggregate: optionally include a t-test score\n>\n> By the way, have you heard about Dumbbench module on CPAN?  It is a\n> module that attempts to implement reasonably robust benchmarking,\n> trying to find and discard bad runs (outliers):\n>\n>   http://search.cpan.org/perldoc?Dumbbench\n>\n> It might be an easier solution (thanks to wonders of local::lib) than\n> GNU R statistical package.\n\nThanks for that pointer, I'll look into it.  I'm less worried about\nwhich particular package to use for the t-test (after all it's a pretty\nwell-known method and could also be hacked in directly), but from a\nquick glance at the TOC they seem to have put a lot of thought into the\npost-processing.\n\n>>   perf: implement a test-selection feature\n>>   perf: add a bisection tool\n>\n> It looks like those last two (10/11, 11/11) didn't made it to git\n> mailing list, probably because of anti-spam size limits of email\n> message on vger.\n\nOops, my bad.  It's not the filter but a 0 too many in my send-email\nglobs.  I sent them out, but for your merging convenience it's also at\n\n  git://github.com/trast/git.git t/perf-gather-post-1.7.10\n\n-- \nThomas Rast\ntrast@{inf,student}.ethz.ch\n"},{"id":"186739","messageId":"87sjhd4w1h.fsf@thomas.inf.ethz.ch","threadId":"29918","inReplyTo":"CACsJy8BQXvX6iOH0txcw5wmW-2ddnG3A39BxsrspfkqPTwDy_g@mail.gmail.com","subject":"Re: [PATCH 00/11] perf improvements past v1.7.10","fromName":"Thomas Rast","fromEmail":"trast@inf.ethz.ch","sentAt":"2012-03-12T16:35:22Z","receivedAt":"2012-03-12T16:35:22Z","isPatch":true,"sender":{"key":"tr@thomasrast.ch","avatar":"https://avatars.githubusercontent.com/u/153510?v=4"},"body":"Nguyen Thai Ngoc Duy <pclouds@gmail.com> writes:\n\n> Not really related to the series, how do I compare the same tests on\n> different repositories? These repositories share the same DAG, but may\n> have different object representation (all loose, single pack, zillion\n> of packs, different versions of pack...)\n\nCurrently the only way is to run the test once for each setting of\nGIT_PERF_REPO.  The only comparisons it does so far is between\nrevisions/build directories.\n\nThat's just because at the time, I saw neither a clear need nor a simple\nsyntax that I could use.  I'd welcome suggestions, however.\n\nMy only use-case that 'run' does not handle so far would just be\nmirroring the table, i.e., getting a comparison of different tests (such\nas 'rebase', 'rebase -m' etc.)  from the same build tree.\n\n-- \nThomas Rast\ntrast@{inf,student}.ethz.ch\n"},{"id":"186748","messageId":"87ipi9yaws.fsf@thomas.inf.ethz.ch","threadId":"29918","inReplyTo":"e6a6af53264368a25e71719901f3227042e21f5c.1331561353.git.trast@student.ethz.ch","subject":"Re: [PATCH 02/11] Introduce a performance test for git-rebase","fromName":"Thomas Rast","fromEmail":"trast@inf.ethz.ch","sentAt":"2012-03-12T17:41:23Z","receivedAt":"2012-03-12T17:41:23Z","isPatch":true,"sender":{"key":"tr@thomasrast.ch","avatar":"https://avatars.githubusercontent.com/u/153510?v=4"},"body":"Thomas Rast <trast@student.ethz.ch> writes:\n\n> The perf framework lets the user run the test in an arbitrary repo,\n> which makes writing a test for rebase a bit finicky.  This one uses a\n> perl script to determine a longest linear segment of history, that is,\n> a range A..B which is completely linear.  (For a full clone of\n> git.git, this is the (whole) 'html' branch.)  The number of commits\n> that are rebased is capped at 50, however.  That is, if A..B is more\n> than 50 commits, it uses B~50..B instead.\n>\n> Having determined a suitable range, it then runs rebase with all\n> combinations of rerere (on/off), -i / noninteractive, and -m / normal.\n>\n> Signed-off-by: Thomas Rast <trast@student.ethz.ch>\n> ---\n\nI forgot to put the interesting part here:\n\n Test                                   this tree\n ------------------------------------------------------\n 3400.4: rebase -f (rerere off)         4.64(2.12+1.80)\n 3400.5: rebase -m -f (rerere off)      4.17(2.07+1.53)  to .4: -10% ***\n 3400.6: rebase -i -f (rerere off)      4.51(2.30+1.57)  to .4:  -3% *\n 3400.7: rebase -i -m -f (rerere off)   4.51(2.31+1.57)\n ------------------------------------------------------\n Significance hints:  '.' 0.1  '*' 0.05  '**' 0.01  '***' 0.001\n\nI did the change/significance manually here, so I might have made a\nmistake in the exact numbers.\n\nHowever, it's pretty obvious that 'rebase -f' is slow compared to\n'rebase -m'.  So slow in fact that the user would be better off running\nan *interactive* rebase.\n\nSo what's the advantage of not using -m?  I always thought it was\nbecause plain rebase was supposed to be *faster* than -m, and I\nconfirmed this in an extremely scientific test on #git-devel:\n\n  [09 Mar 14:05] <charon> hrm, what's the advantage of straight rebase over 'rebase -m' ?\n  [09 Mar 14:06] <mhagger> charon: It's supposed to be significantly faster\n  [09 Mar 14:06] <charon> mhagger: ah thanks, that's what i suspected\n\nThis wasn't always the case; back in v1.7.0 you would see this picture:\n\n Test                                v1.7.0\n ---------------------------------------------------\n 3400.4: rebase -f (rerere off)      3.75(2.12+1.18)\n 3400.5: rebase -m -f (rerere off)   4.05(2.29+1.32)  to .4: +8% ***\n\nThis comes from the i18n support added in adc3b2b27..a9f578685, and from\n43c2325 (am: use get_author_ident_from_commit instead of mailinfo when\nrebasing, 2010-06-16).\n\nNow I'm not really keen on hacking on git-am until it gets back its\nperformance, as for most uses it's probably still fast enough.  But I\nthink one important question is: can we get rid of the\ngit-format-patch|git-am mode of git-rebase, which has historically been\na source of pain (see, e.g., the aforementioned 43c2325)?\n\nOk, perhaps not: certain features like --whitespace depend on it :-(\n\n--\nThomas Rast\ntrast@{inf,student}.ethz.ch\n"},{"id":"186768","messageId":"7v399dsiw2.fsf@alter.siamese.dyndns.org","threadId":"29918","inReplyTo":"87ipi9yaws.fsf@thomas.inf.ethz.ch","subject":"Re: [PATCH 02/11] Introduce a performance test for git-rebase","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2012-03-12T19:45:33Z","receivedAt":"2012-03-12T19:45:33Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Thomas Rast <trast@inf.ethz.ch> writes:\n\n> However, it's pretty obvious that 'rebase -f' is slow compared to\n> 'rebase -m'.  So slow in fact that the user would be better off running\n> an *interactive* rebase.\n\nThe mention of \"-f\" got me off track, but does \"-m\" avoid creating\nnew commits when it does not have to?  A side effect \"-f\" has is to\nreplay _all_ the commits, if I recall correctly.\n\n> This comes from the i18n support added in adc3b2b27..a9f578685, and from\n> 43c2325 (am: use get_author_ident_from_commit instead of mailinfo when\n> rebasing, 2010-06-16).\n\n> Now I'm not really keen on hacking on git-am until it gets back its\n> performance, as for most uses it's probably still fast enough.  But I\n> think one important question is: can we get rid of the\n> git-format-patch|git-am mode of git-rebase, which has historically been\n> a source of pain (see, e.g., the aforementioned 43c2325)?\n\nIn the longer term, the right way forward is to try doing less in\nthe patch mode in rebase.  For example, the command could make the\nlist of commits to be replayed exactly the same way as it does for\nflattening cherry-pick rebase, and instead of running \"am\", run\n\"format-patch --stdout | apply --3way --whitespace=fix --index\" and\ncommit the result using \"commit -C\".  That way it can depend on\n\"commit -C\" and does not have to worry about stupid things like\n43c2325.\n"},{"id":"186777","messageId":"87obs1po5a.fsf@thomas.inf.ethz.ch","threadId":"29918","inReplyTo":"7v399dsiw2.fsf@alter.siamese.dyndns.org","subject":"Re: [PATCH 02/11] Introduce a performance test for git-rebase","fromName":"Thomas Rast","fromEmail":"trast@inf.ethz.ch","sentAt":"2012-03-12T20:20:17Z","receivedAt":"2012-03-12T20:20:17Z","isPatch":true,"sender":{"key":"tr@thomasrast.ch","avatar":"https://avatars.githubusercontent.com/u/153510?v=4"},"body":"Junio C Hamano <gitster@pobox.com> writes:\n\n> Thomas Rast <trast@inf.ethz.ch> writes:\n>\n>> However, it's pretty obvious that 'rebase -f' is slow compared to\n>> 'rebase -m'.  So slow in fact that the user would be better off running\n>> an *interactive* rebase.\n>\n> The mention of \"-f\" got me off track, but does \"-m\" avoid creating\n> new commits when it does not have to?  A side effect \"-f\" has is to\n> replay _all_ the commits, if I recall correctly.\n\nSorry, my bad.  *All* tests use -f for exactly that reason.\n\n>> Now I'm not really keen on hacking on git-am until it gets back its\n>> performance, as for most uses it's probably still fast enough.  But I\n>> think one important question is: can we get rid of the\n>> git-format-patch|git-am mode of git-rebase, which has historically been\n>> a source of pain (see, e.g., the aforementioned 43c2325)?\n>\n> In the longer term, the right way forward is to try doing less in\n> the patch mode in rebase.  For example, the command could make the\n> list of commits to be replayed exactly the same way as it does for\n> flattening cherry-pick rebase, and instead of running \"am\", run\n> \"format-patch --stdout | apply --3way --whitespace=fix --index\" and\n> commit the result using \"commit -C\".  That way it can depend on\n> \"commit -C\" and does not have to worry about stupid things like\n> 43c2325.\n\nAh, but that's a trap, or do you have an apply --3way series sitting\nsomewhere ;-)\n\n-- \nThomas Rast\ntrast@{inf,student}.ethz.ch\n"},{"id":"186782","messageId":"7v1uoxr0vs.fsf@alter.siamese.dyndns.org","threadId":"29918","inReplyTo":"87obs1po5a.fsf@thomas.inf.ethz.ch","subject":"Re: [PATCH 02/11] Introduce a performance test for git-rebase","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2012-03-12T20:59:51Z","receivedAt":"2012-03-12T20:59:51Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Thomas Rast <trast@inf.ethz.ch> writes:\n\n> Ah, but that's a trap, or do you have an apply --3way series sitting\n> somewhere ;-)\n\nAccording to rumors, it is more trivial than a typical summer\nproject for a GSoC student, so...\n"}]}