{"thread":{"id":"28771","subject":"Is there a place for benchmarking scripts?","startedAt":"2011-10-26T09:50:27Z","lastAt":"2011-10-27T16:01:40Z","messageCount":2,"participants":["Michael Haggerty","René Scharfe"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"178312","messageId":"4EA7D7E3.2020009@alum.mit.edu","threadId":"28771","inReplyTo":null,"subject":"Is there a place for benchmarking scripts?","fromName":"Michael Haggerty","fromEmail":"mhagger@alum.mit.edu","sentAt":"2011-10-26T09:50:27Z","receivedAt":"2011-10-26T09:50:27Z","isPatch":false,"sender":{"key":"mhagger@alum.mit.edu","avatar":"https://avatars.githubusercontent.com/u/119718?v=4"},"body":"I've been doing a lot of benchmarking of git performance in the presence\nof lots of references. I've written a few scripts to automate the\nbenchmarking [1]. They are not beautiful and would require a couple of\nlocal adjustments [2,3]. They are too time-consuming to be made part of\nthe usual test suite. I wouldn't want to commit to maintaining them.\nBut they have certainly been useful to me, and they generate readable\noutput [4].\n\nMy question is: would such benchmarking scripts be welcome within the\ngit project? If so, where should I put it? Is any benchmarking\ncode/framework already in use?\n\nMichael\n\n[1] Branch \"refperf\" at git://github.com/mhagger/git.git\n\n[2] For example, one script checks out specified git revisions, merges\nin the \"refperf\" branch to get the benchmarking code, then runs tests.\nThis script has to be adjusted with the local name of the \"refperf\"\nbranch (e.g., \"refperf\" vs. \"origin/refperf\" vs. ...)\n\n[3] I wanted the tests to include cases with a cold disk cache and with\na warm disk cache. For the former, I have the script run \"sync; sudo sh\n-c 'echo 3 >/proc/sys/vm/drop_caches'\" which obviously only works on\nLinux and only when the user has password-less sudo permissions.\n\n[4] Example (numbers are times in seconds):\n\n=================================== ======== ======== ========\nTest name [0] [1] [2]\n=================================== ======== ======== ========\nbranch-loose-cold 3.32 3.25 0.55\nbranch-loose-warm 0.60 0.22 0.00\nfor-each-ref-loose-cold 3.71 3.46 3.45\nfor-each-ref-loose-warm 0.83 0.46 0.47\ncheckout-loose-cold 3.82 3.23 0.63\ncheckout-loose-warm 0.58 0.21 0.01\ncheckout-orphan-loose 0.58 0.20 0.00\ncheckout-from-detached-loose-cold 4.54 4.20 3.75\ncheckout-from-detached-loose-warm 1.41 1.07 0.63\nbranch-contains-loose-cold 4.04 3.71 3.67\nbranch-contains-loose-warm 0.93 0.57 0.56\npack-refs-loose 2.31 1.92 1.91\nbranch-packed-cold 0.53 0.50 0.49\nbranch-packed-warm 0.02 0.02 0.03\nfor-each-ref-packed-cold 1.01 0.92 0.94\nfor-each-ref-packed-warm 0.26 0.27 0.28\ncheckout-packed-cold 14.37 1.51 1.27\ncheckout-packed-warm 0.04 0.03 0.03\ncheckout-orphan-packed 0.02 0.02 0.03\ncheckout-from-detached-packed-cold 14.54 1.51 1.12\ncheckout-from-detached-packed-warm 13.85 0.82 0.46\nbranch-contains-packed-cold 1.03 1.04 1.01\nbranch-contains-packed-warm 0.37 0.38 0.39\nclone-loose-cold 13.17 11.97 12.45\nclone-loose-warm 6.86 5.40 5.83\nfetch-nothing-loose 1.24 1.11 1.52\npack-refs 0.27 0.27 0.29\nfetch-nothing-packed 1.23 1.11 1.52\nclone-packed-cold 1.89 1.84 1.80\nclone-packed-warm 0.62 0.63 0.70\nfetch-everything-cold 12.85 13.07 13.40\nfetch-everything-warm 6.53 6.92 6.88\nfilter-branch-warm 35383.91 15869.38 748.01\n=================================== ======== ======== ========\n\n\n[0] 703f05a (tag: v1.7.7) Git 1.7.7\n Test repository created using: t/make-refperf-repo --commits 20000\n--refs 10000 --shard\n[1] 2c5c66b Merge branch 'jp/get-ref-dir-unsorted'\n Test repository created using: t/make-refperf-repo --commits 20000\n--refs 10000 --shard\n[2] f579019 (ref-api-D) sort_ref_dir(): remove the recurse argument\n Test repository created using: t/make-refperf-repo --commits 20000\n--refs 10000 --shard\n\n\n-- \nMichael Haggerty\nmhagger@alum.mit.edu\nhttp://softwareswirl.blogspot.com/\n"},{"id":"178355","messageId":"4EA98064.3060706@lsrfire.ath.cx","threadId":"28771","inReplyTo":"4EA7D7E3.2020009@alum.mit.edu","subject":"Re: Is there a place for benchmarking scripts?","fromName":"René Scharfe","fromEmail":"rene.scharfe@lsrfire.ath.cx","sentAt":"2011-10-27T16:01:40Z","receivedAt":"2011-10-27T16:01:40Z","isPatch":false,"sender":{"key":"l.s.r@web.de","avatar":"https://avatars.githubusercontent.com/u/26122331?v=4"},"body":"Am 26.10.2011 11:50, schrieb Michael Haggerty:\n> I've been doing a lot of benchmarking of git performance in the presence\n> of lots of references. I've written a few scripts to automate the\n> benchmarking [1]. They are not beautiful and would require a couple of\n> local adjustments [2,3]. They are too time-consuming to be made part of\n> the usual test suite. I wouldn't want to commit to maintaining them.\n> But they have certainly been useful to me, and they generate readable\n> output [4].\n> \n> My question is: would such benchmarking scripts be welcome within the\n> git project? If so, where should I put it? Is any benchmarking\n> code/framework already in use?\n\nThat would be nice. A whole performance regression testing suite would\nbe even nicer and can perhaps be built piece by piece.\n\nWe have contrib/ and we have the test-* commands; t/ doesn't seem to fit\ntoo well with its focus on OK or fail.\n\nRené\n"}]}