{"thread":{"id":"27898","subject":"[RFC PATCH] tests: print failed test numbers at the end of the test run","startedAt":"2011-07-23T19:16:02Z","lastAt":"2011-07-26T06:10:38Z","messageCount":9,"participants":["Jens Lehmann","Junio C Hamano","Jeff King","Sverre Rabbelier","Nguyen Thai Ngoc Duy"],"isPatch":true,"patchVersion":1,"patchTotal":null},"messages":[{"id":"171898","messageId":"4E2B1DF2.4000003@web.de","threadId":"27898","inReplyTo":null,"subject":"[RFC PATCH] tests: print failed test numbers at the end of the test run","fromName":"Jens Lehmann","fromEmail":"jens.lehmann@web.de","sentAt":"2011-07-23T19:16:02Z","receivedAt":"2011-07-23T19:16:02Z","isPatch":true,"sender":{"key":"jens.lehmann@web.de","avatar":"https://avatars.githubusercontent.com/u/135220?v=4"},"body":"On modern multi-core processors \"make test\" is often run in multiple jobs.\nIf one of them fails the test run does stop, but the concurrently running\ntests finish their run. Finding out what test is broken involves a lot of\nscrolling. That gets even worse when the -i option is used.\n\nIf one or more tests failed, print a list of them before the test summary:\n\nfailed test(s): t1000 t6500\n\nfixed 0\nsuccess 7638\nfailed 3\nbroken 49\ntotal 7723\n\nThis makes it possible to just run the test suite with -i and collect all\nfailed test scripts at the end for further examination.\n\nSigned-off-by: Jens Lehmann \n---\n\nMaybe I'm missing something completely obvious, but I always have a hard\ntime finding out which test scripts did fail in a test run with -j30.\n\n t/aggregate-results.sh | 12 +++++++++++-\n 1 files changed, 11 insertions(+), 1 deletions(-)\n\ndiff --git a/t/aggregate-results.sh b/t/aggregate-results.sh\nindex d206b7c..b8e929a 100755\n--- a/t/aggregate-results.sh\n+++ b/t/aggregate-results.sh\n@@ -1,5 +1,6 @@\n #!/bin/sh\n\n+failed_tests=\n fixed=0\n success=0\n failed=0\n@@ -18,7 +19,12 @@ do\n \t\tsuccess)\n \t\t\tsuccess=$(($success + $value)) ;;\n \t\tfailed)\n-\t\t\tfailed=$(($failed + $value)) ;;\n+\t\t\tfailed=$(($failed + $value))\n+\t\t\tif test $value != 0; then\n+\t\t\t\ttestnum=$(echo $file | cut -b 14-18)\n+\t\t\t\tfailed_tests=\"$failed_tests $testnum\"\n+\t\t\tfi\n+\t\t\t;;\n \t\tbroken)\n \t\t\tbroken=$(($broken + $value)) ;;\n \t\ttotal)\n@@ -27,6 +33,10 @@ do\n \tdone <\"$file\"\n done\n\n+if [ -n \"$failed_tests\" ]; then\n+\tprintf \"\\nfailed test(s):$failed_tests\\n\\n\"\n+fi\n+\n printf \"%-8s%d\\n\" fixed $fixed\n printf \"%-8s%d\\n\" success $success\n printf \"%-8s%d\\n\" failed $failed\n-- \n1.7.6.346.g750efc\n"},{"id":"171924","messageId":"7vmxg4w2fm.fsf@alter.siamese.dyndns.org","threadId":"27898","inReplyTo":"4E2B1DF2.4000003@web.de","subject":"Re: [RFC PATCH] tests: print failed test numbers at the end of the test run","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2011-07-24T07:41:33Z","receivedAt":"2011-07-24T07:41:33Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Jens Lehmann writes:\n\n> Maybe I'm missing something completely obvious, but I always have a hard\n> time finding out which test scripts did fail in a test run with -j30.\n\nI run \"ls -d t/trash*\" for that ;-)\n\n> +\t\t\tif test $value != 0; then\n> +\t\t\t\ttestnum=$(echo $file | cut -b 14-18)\n> +\t\t\t\tfailed_tests=\"$failed_tests $testnum\"\n> +\t\t\tfi\n\nSomehow \"cut\" here feels dirty, as it hardcodes a mysterious offset that\ncannot be guessed without knowing what string it is cutting from by\nlooking at t/Makefile.\n\n\ttestnum=$(expr \"$file\" : 'test-results/\\(t[0-9]*\\)-')\n"},{"id":"171943","messageId":"4E2C1FBA.3010606@web.de","threadId":"27898","inReplyTo":"7vmxg4w2fm.fsf@alter.siamese.dyndns.org","subject":"[PATCH v2] tests: print failed test numbers at the end of the test run","fromName":"Jens Lehmann","fromEmail":"jens.lehmann@web.de","sentAt":"2011-07-24T13:35:54Z","receivedAt":"2011-07-24T13:35:54Z","isPatch":true,"sender":{"key":"jens.lehmann@web.de","avatar":"https://avatars.githubusercontent.com/u/135220?v=4"},"body":"On modern multi-core processors \"make test\" is often run in multiple jobs.\nIf one of them fails the test run does stop, but the concurrently running\ntests finish their run. It is rather easy to find out which test failed by\ndoing a \"ls -d t/trash*\". But that only works when you don't use the \"-i\"\noption to \"make test\" because you want to get an overview of all failing\ntests. In that case all thrash directories are deleted end and the\ninformation which tests failed is lost.\n\nIf one or more tests failed, print a list of them before the test summary:\n\nfailed test(s): t1000 t6500\n\nfixed 0\nsuccess 7638\nfailed 3\nbroken 49\ntotal 7723\n\nThis makes it possible to just run the test suite with -i and collect all\nfailed test scripts at the end for further examination.\n\nSigned-off-by: Jens Lehmann \n---\n\nAm 24.07.2011 09:41, schrieb Junio C Hamano:\n> Jens Lehmann writes:\n> \n>> Maybe I'm missing something completely obvious, but I always have a hard\n>> time finding out which test scripts did fail in a test run with -j30.\n> \n> I run \"ls -d t/trash*\" for that ;-)\n\nYup. But that won't work when you use \"-i\" to continue to run all tests\neven when some fail, as all trash directories get deleted in the end. Or\ndid I manage to overlook an option which stops that? ;-)\n\nI rephrased the commit message to make it clear my patch only changes the\nbehavior when \"-i\" is used and one or more tests fail, as the code in\nquestion isn't executed otherwise.\n\n>> +\t\t\tif test $value != 0; then\n>> +\t\t\t\ttestnum=$(echo $file | cut -b 14-18)\n>> +\t\t\t\tfailed_tests=\"$failed_tests $testnum\"\n>> +\t\t\tfi\n> \n> Somehow \"cut\" here feels dirty, as it hardcodes a mysterious offset that\n> cannot be guessed without knowing what string it is cutting from by\n> looking at t/Makefile.\n> \n> \ttestnum=$(expr \"$file\" : 'test-results/\\(t[0-9]*\\)-')\n\nThanks, this version is using that expression.\n\n\n t/aggregate-results.sh | 12 +++++++++++-\n 1 files changed, 11 insertions(+), 1 deletions(-)\n\ndiff --git a/t/aggregate-results.sh b/t/aggregate-results.sh\nindex d206b7c..60465fa 100755\n--- a/t/aggregate-results.sh\n+++ b/t/aggregate-results.sh\n@@ -1,5 +1,6 @@\n #!/bin/sh\n\n+failed_tests=\n fixed=0\n success=0\n failed=0\n@@ -18,7 +19,12 @@ do\n \t\tsuccess)\n \t\t\tsuccess=$(($success + $value)) ;;\n \t\tfailed)\n-\t\t\tfailed=$(($failed + $value)) ;;\n+\t\t\tfailed=$(($failed + $value))\n+\t\t\tif test $value != 0; then\n+\t\t\t\ttestnum=$(expr \"$file\" : 'test-results/\\(t[0-9]*\\)-')\n+\t\t\t\tfailed_tests=\"$failed_tests $testnum\"\n+\t\t\tfi\n+\t\t\t;;\n \t\tbroken)\n \t\t\tbroken=$(($broken + $value)) ;;\n \t\ttotal)\n@@ -27,6 +33,10 @@ do\n \tdone <\"$file\"\n done\n\n+if [ -n \"$failed_tests\" ]; then\n+\tprintf \"\\nfailed test(s):$failed_tests\\n\\n\"\n+fi\n+\n printf \"%-8s%d\\n\" fixed $fixed\n printf \"%-8s%d\\n\" success $success\n printf \"%-8s%d\\n\" failed $failed\n-- \n1.7.6.346.g940def.dirty\n"},{"id":"171960","messageId":"20110724153950.GB7805@sigill.intra.peff.net","threadId":"27898","inReplyTo":"4E2C1FBA.3010606@web.de","subject":"Re: [PATCH v2] tests: print failed test numbers at the end of the test run","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2011-07-24T15:39:51Z","receivedAt":"2011-07-24T15:39:51Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Sun, Jul 24, 2011 at 03:35:54PM +0200, Jens Lehmann wrote:\n\n> Am 24.07.2011 09:41, schrieb Junio C Hamano:\n> > Jens Lehmann writes:\n> > \n> >> Maybe I'm missing something completely obvious, but I always have a hard\n> >> time finding out which test scripts did fail in a test run with -j30.\n> > \n> > I run \"ls -d t/trash*\" for that ;-)\n> \n> Yup. But that won't work when you use \"-i\" to continue to run all tests\n> even when some fail, as all trash directories get deleted in the end. Or\n> did I manage to overlook an option which stops that? ;-)\n\nI used to use:\n\n make -k -j30\n grep 'failed [^0]' test-results/*\n\nbut these days I use the \"prove\" tool, which has much nicer output. You\ncan get Makefile support for it with:\n\n GIT_PROVE_OPTS = -j30\n DEFAULT_TEST_TARGET = prove\n\nin your config.mak.\n\n-Peff\n"},{"id":"171967","messageId":"4E2C70AB.2060306@web.de","threadId":"27898","inReplyTo":"20110724153950.GB7805@sigill.intra.peff.net","subject":"Re: [PATCH v2] tests: print failed test numbers at the end of the test run","fromName":"Jens Lehmann","fromEmail":"jens.lehmann@web.de","sentAt":"2011-07-24T19:21:15Z","receivedAt":"2011-07-24T19:21:15Z","isPatch":true,"sender":{"key":"jens.lehmann@web.de","avatar":"https://avatars.githubusercontent.com/u/135220?v=4"},"body":"Am 24.07.2011 17:39, schrieb Jeff King:\n> On Sun, Jul 24, 2011 at 03:35:54PM +0200, Jens Lehmann wrote:\n> \n>> Am 24.07.2011 09:41, schrieb Junio C Hamano:\n>>> Jens Lehmann writes:\n>>>\n>>>> Maybe I'm missing something completely obvious, but I always have a hard\n>>>> time finding out which test scripts did fail in a test run with -j30.\n>>>\n>>> I run \"ls -d t/trash*\" for that ;-)\n>>\n>> Yup. But that won't work when you use \"-i\" to continue to run all tests\n>> even when some fail, as all trash directories get deleted in the end. Or\n>> did I manage to overlook an option which stops that? ;-)\n> \n> I used to use:\n> \n> make -k -j30\n> grep 'failed [^0]' test-results/*\n\nAh, now I understand how I managed to shoot myself in the foot when I tried\nthat: I used \"-i\" instead of \"-k\" for make, so the test-results directory\nalways was deleted after the tests ran.\n\n> but these days I use the \"prove\" tool, which has much nicer output. You\n> can get Makefile support for it with:\n> \n> GIT_PROVE_OPTS = -j30\n> DEFAULT_TEST_TARGET = prove\n> \n> in your config.mak.\n\nThanks, the output really is nicer and it even runs a bit faster! :-)\n"},{"id":"171990","messageId":"CAGdFq_gCQwgk0yN5e-DJWcURx02ETT_8KD3qGw4hBnMg8PbWTw@mail.gmail.com","threadId":"27898","inReplyTo":"4E2C70AB.2060306@web.de","subject":"Re: [PATCH v2] tests: print failed test numbers at the end of the test run","fromName":"Sverre Rabbelier","fromEmail":"srabbelier@gmail.com","sentAt":"2011-07-25T14:31:36Z","receivedAt":"2011-07-25T14:31:36Z","isPatch":true,"sender":{"key":"srabbelier@gmail.com","avatar":"https://avatars.githubusercontent.com/u/3098?v=4"},"body":"Heya,\n\nOn Sun, Jul 24, 2011 at 21:21, Jens Lehmann wrote:\n> Ah, now I understand how I managed to shoot myself in the foot when I tried\n> that: I used \"-i\" instead of \"-k\" for make, so the test-results directory\n> always was deleted after the tests ran.\n\nRegardless, I like your patch. I don't use prove, and I'm always\nafraid that when I run the test suite I'll find a test that failed\nwith no way to find out which :).\n\n-- \nCheers,\n\nSverre Rabbelier\n"},{"id":"172033","messageId":"CACsJy8CuqHSoG4PVMiT=5EUv2hn=42y5B+6Rhny2VkXW32WR8Q@mail.gmail.com","threadId":"27898","inReplyTo":"4E2B1DF2.4000003@web.de","subject":"Re: [RFC PATCH] tests: print failed test numbers at the end of the test run","fromName":"Nguyen Thai Ngoc Duy","fromEmail":"pclouds@gmail.com","sentAt":"2011-07-26T04:25:21Z","receivedAt":"2011-07-26T04:25:21Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"On Sun, Jul 24, 2011 at 2:16 AM, Jens Lehmann wrote:\n> On modern multi-core processors \"make test\" is often run in multiple jobs.\n> If one of them fails the test run does stop, but the concurrently running\n> tests finish their run.\n\nSomewhat related (or not). I change something. I know it breaks things\nand want to know _all_ tests it breaks, but \"make test\" would stop\nearly. Is there anyway to make it keep going through all tests even if\nsome fails? \"make -j\" improves the situation but does not\nreally solve it.\n-- \nDuy\n"},{"id":"172036","messageId":"20110726054236.GC25046@sigill.intra.peff.net","threadId":"27898","inReplyTo":"CACsJy8CuqHSoG4PVMiT=5EUv2hn=42y5B+6Rhny2VkXW32WR8Q@mail.gmail.com","subject":"Re: [RFC PATCH] tests: print failed test numbers at the end of the test run","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2011-07-26T05:42:36Z","receivedAt":"2011-07-26T05:42:36Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Tue, Jul 26, 2011 at 11:25:21AM +0700, Nguyen Thai Ngoc Duy wrote:\n\n> On Sun, Jul 24, 2011 at 2:16 AM, Jens Lehmann wrote:\n> > On modern multi-core processors \"make test\" is often run in multiple jobs.\n> > If one of them fails the test run does stop, but the concurrently running\n> > tests finish their run.\n> \n> Somewhat related (or not). I change something. I know it breaks things\n> and want to know _all_ tests it breaks, but \"make test\" would stop\n> early. Is there anyway to make it keep going through all tests even if\n> some fails? \"make -j\" improves the situation but does not\n> really solve it.\n\nTry \"make -k\", which will keep running rules that don't have failed\ndependencies (in this case, all of the tests are independent, so it will\nrun all of them).\n\nOr use \"prove\", which can do fancy things like putting the last-failed\n(so you can see the likely failures immediately and start to probe them\nwhile the rest of the suite runs).\n\n-Peff\n"},{"id":"172039","messageId":"20110726061038.GB29486@sigill.intra.peff.net","threadId":"27898","inReplyTo":"20110726054236.GC25046@sigill.intra.peff.net","subject":"Re: [RFC PATCH] tests: print failed test numbers at the end of the test run","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2011-07-26T06:10:38Z","receivedAt":"2011-07-26T06:10:38Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Mon, Jul 25, 2011 at 11:42:36PM -0600, Jeff King wrote:\n\n> Or use \"prove\", which can do fancy things like putting the last-failed\n> (so you can see the likely failures immediately and start to probe them\n> while the rest of the suite runs).\n\nOops, that should read \"...putting the last-failed tests at the\nbeginning\".\n\n-Peff\n"}]}