{"thread":{"id":"29893","subject":"[PATCH] p4000: use -3000 when promising -3000","startedAt":"2012-03-09T09:52:54Z","lastAt":"2012-03-09T09:52:54Z","messageCount":1,"participants":["Thomas Rast"],"isPatch":true,"patchVersion":1,"patchTotal":null},"messages":[{"id":"186550","messageId":"16d90c224b9a0cac0054f8903b9c674a67270cda.1331285117.git.trast@student.ethz.ch","threadId":"29893","inReplyTo":null,"subject":"[PATCH] p4000: use -3000 when promising -3000","fromName":"Thomas Rast","fromEmail":"trast@student.ethz.ch","sentAt":"2012-03-09T09:52:54Z","receivedAt":"2012-03-09T09:52:54Z","isPatch":true,"sender":{"key":"tr@thomasrast.ch","avatar":"https://avatars.githubusercontent.com/u/153510?v=4"},"body":"The 'log -3000 (baseline)' test accidentally still used -1000 from an\nearlier version.\n\nNoticed-by: Lawrence Holding <Lawrence.Holding@cubic.com>\nSigned-off-by: Thomas Rast <trast@student.ethz.ch>\n---\n\nThat has to be the worst paperbag bug in quite some time, especially\nconsidering that this is already in master and -rc0.  Luckily it\ndoesn't affect the validity of the *actual* tests, and thus what the\ncommit message said.\n\n\nGoing off on a tangent: I played with bisecting for performance\nregressions yesterday, and this shows a bit of a problem with all of\nthese tests.\n\nI could use an unrestricted 'log' here, but then the perf suite would\nbe too slow to \"just run\" with defaults (i.e., on git.git).  For\nexample, if I run p4000 with 'sed -i s/-3000//' applied I get\n\n  Test                             this tree        \n  --------------------------------------------------\n  4000.1: log  (baseline)          0.59(0.52+0.06)  \n  4000.2: log --raw  (tree-only)   3.02(2.57+0.43)  \n  4000.3: log -p  (Myers)          17.66(16.65+0.93)\n  4000.4: log -p  --histogram      17.64(16.78+0.79)\n  4000.5: log -p  --patience       20.71(19.68+0.96)\n\nFor best-of-3 runs (the default) that's still a runtime on the order\nof 3min -- for a single test!\n\nThe -3000 is also plenty, on my box, to prove (statistically, with\nsignificance to spare) an improvement with this hack for the diff\nengine:\n\n  https://github.com/trast/git/commit/a0acba9b853ae257260fd7bd2f1a81389e1c75e8\n\nOTOH I had some problems with completely unattended bisections.\nConsider the results, for the fixed version:\n\n  Test                                  this tree      \n  -----------------------------------------------------\n  4000.1: log -3000 (baseline)          0.08(0.07+0.01)\n  4000.2: log --raw -3000 (tree-only)   0.38(0.33+0.04)\n  4000.3: log -p -3000 (Myers)          1.70(1.58+0.11)\n  4000.4: log -p -3000 --histogram      1.67(1.57+0.09)\n  4000.5: log -p -3000 --patience       2.02(1.89+0.12)\n\nSuppose I was bisecting for a 5% performance hit.  On the 1.70s\ntimings, that would be about 90ms, but the noise in the timings can\neasily be 30-50ms.  So there is a gap between \"nice for humans to just\nrun\" and \"great S/N\".  \"Reasonable S/N\" won't do it for a bisection,\nas a single bad decision sends you the wrong way.\n\nI did, however, get good results by removing the -3000 limits and then\nautomating a bisection for an 8% performance hit in 4000.3 between\nv1.7.0 and master, which turns out to be explained by 3e5a188 (diff.c:\nEnsure \"index $from..$to\" line contains unambiguous SHA1s,\n2010-05-30).\n\nSo perhaps there should be a knob that says \"and now take 10 times\nlonger please\".  Or something like that.\n\n\n t/perf/p4000-diff-algorithms.sh |    2 +-\n 1 file changed, 1 insertion(+), 1 deletion(-)\n\ndiff --git a/t/perf/p4000-diff-algorithms.sh b/t/perf/p4000-diff-algorithms.sh\nindex d6e505c..7e00c9d 100755\n--- a/t/perf/p4000-diff-algorithms.sh\n+++ b/t/perf/p4000-diff-algorithms.sh\n@@ -7,7 +7,7 @@ test_description=\"Tests diff generation performance\"\n test_perf_default_repo\n \n test_perf 'log -3000 (baseline)' '\n-\tgit log -1000 >/dev/null\n+\tgit log -3000 >/dev/null\n '\n \n test_perf 'log --raw -3000 (tree-only)' '\n-- \n1.7.10.rc0.218.g3ed6ff\n"}]}