{"thread":{"id":"13861","subject":"[PATCH 3/3] Hook up the result aggregation in the test makefile.","startedAt":"2008-06-08T14:04:33Z","lastAt":"2008-06-09T00:44:01Z","messageCount":9,"participants":["Sverre Rabbelier","Jakub Narebski","Junio C Hamano"],"isPatch":true,"patchVersion":1,"patchTotal":3},"messages":[{"id":"79111","messageId":"1212933875-29947-1-git-send-email-sverre@rabbelier.nl","threadId":"13861","inReplyTo":null,"subject":"[PATCH 1/3] Modified test-lib.sh to output stats to /tmp/git-test-results","fromName":"Sverre Rabbelier","fromEmail":"sverre@rabbelier.nl","sentAt":"2008-06-08T14:04:33Z","receivedAt":"2008-06-08T14:04:33Z","isPatch":true,"sender":{"key":"sverre@rabbelier.nl","avatar":"https://gravatar.com/avatar/a66bc99b1c608811dca5c1855e2b55f030478a256cf2c947eee52c0404e710bf?d=mp&s=160"},"body":"From: Sverre Rabbelier \n\nThis change is needed order to aggregate data on the test run later on.\nBecause writing to the current directory is not possible, we write to /tmp/.\nSuggestions for a better location are welcome.\n\nSigned-off-by: Sverre Rabbelier \n---\n t/test-lib.sh | 10 ++++++++++\n 1 files changed, 10 insertions(+), 0 deletions(-)\n\ndiff --git a/t/test-lib.sh b/t/test-lib.sh\nindex 7a8bd27..4585fde 100644\n--- a/t/test-lib.sh\n+++ b/t/test-lib.sh\n@@ -152,6 +152,7 @@ test_failure=0\n test_count=0\n test_fixed=0\n test_broken=0\n+test_success=0\n \n die () {\n \techo >&5 \"FATAL: Unexpected exit with code $?\"\n@@ -193,6 +194,7 @@ test_tick () {\n \n test_ok_ () {\n \ttest_count=$(expr \"$test_count\" + 1)\n+\ttest_success=$(expr \"$test_success\" + 1)\n \tsay_color \"\" \" ok $test_count: $@\"\n }\n \n@@ -353,6 +355,14 @@ test_create_repo () {\n \n test_done () {\n \ttrap - exit\n+\ttest_results_path=\"../test-results\"\n+\n+\techo \"total $test_count\" >> $test_results_path\n+\techo \"success $test_success\" >> $test_results_path\n+\techo \"fixed $test_fixed\" >> $test_results_path\n+\techo \"broken $test_broken\" >> $test_results_path\n+\techo \"failed $test_failure\" >> $test_results_path\n+\techo \"\" >> $test_results_path\n \n \tif test \"$test_fixed\" != 0\n \tthen\n-- \n1.5.5.1.214.g82ce2.dirty\n"},{"id":"79112","messageId":"1212933875-29947-2-git-send-email-sverre@rabbelier.nl","threadId":"13861","inReplyTo":"1212933875-29947-1-git-send-email-sverre@rabbelier.nl","subject":"[PATCH 2/3] A simple script to parse the results from the testcases","fromName":"Sverre Rabbelier","fromEmail":"sverre@rabbelier.nl","sentAt":"2008-06-08T14:04:34Z","receivedAt":"2008-06-08T14:04:34Z","isPatch":true,"sender":{"key":"sverre@rabbelier.nl","avatar":"https://gravatar.com/avatar/a66bc99b1c608811dca5c1855e2b55f030478a256cf2c947eee52c0404e710bf?d=mp&s=160"},"body":"From: Miklos Vajna \n\nThis is a simple script that aggregates key:value pairs in a file.\n\nSigned-off-by: Miklos Vajna \n---\n t/aggregate-results.sh | 33 +++++++++++++++++++++++++++++++++\n 1 files changed, 33 insertions(+), 0 deletions(-)\n create mode 100755 t/aggregate-results.sh\n\ndiff --git a/t/aggregate-results.sh b/t/aggregate-results.sh\nnew file mode 100755\nindex 0000000..9e4322c\n--- /dev/null\n+++ b/t/aggregate-results.sh\n@@ -0,0 +1,33 @@\n+#!/bin/sh\n+\n+input=\"test-results\"\n+\n+fixed=0\n+success=0\n+failed=0\n+broken=0\n+total=0\n+\n+while read type value\n+do\n+\tcase $type in\n+\t'')\n+\t\tcontinue ;;\n+\tfixed)\t\n+\t\tfixed=$(($fixed + $value)) ;;\n+\tsuccess)\t\n+\t\tsuccess=$(($success + $value)) ;;\n+\tfailed)\t\n+\t\tfailed=$(($failed + $value)) ;;\n+\tbroken)\t\n+\t\tbroken=$(( $broken + $value)) ;;\n+\ttotal)\t\n+\t\ttotal=$(( $total + $value)) ;;\n+\tesac\n+done < $input\n+\n+printf \"%-8s%d\\n\" fixed $fixed\n+printf \"%-8s%d\\n\" success $success\n+printf \"%-8s%d\\n\" failed $failed\n+printf \"%-8s%d\\n\" broken $broken\n+printf \"%-8s%d\\n\" total $total\n-- \n1.5.5.1.214.g82ce2.dirty\n"},{"id":"79110","messageId":"1212933875-29947-3-git-send-email-sverre@rabbelier.nl","threadId":"13861","inReplyTo":"1212933875-29947-1-git-send-email-sverre@rabbelier.nl","subject":"[PATCH 3/3] Hook up the result aggregation in the test makefile.","fromName":"Sverre Rabbelier","fromEmail":"sverre@rabbelier.nl","sentAt":"2008-06-08T14:04:35Z","receivedAt":"2008-06-08T14:04:35Z","isPatch":true,"sender":{"key":"sverre@rabbelier.nl","avatar":"https://gravatar.com/avatar/a66bc99b1c608811dca5c1855e2b55f030478a256cf2c947eee52c0404e710bf?d=mp&s=160"},"body":"From: Sverre Rabbelier \n\nThis patch makes 'make' output the aggregated results at the end of each build.\nThe 'git-test-result' file is removed both before and after each build.\n\nSigned-off-by: Sverre Rabbelier \n---\n t/Makefile | 11 +++++++++--\n 1 files changed, 9 insertions(+), 2 deletions(-)\n\ndiff --git a/t/Makefile b/t/Makefile\nindex c6a60ab..f9ff933 100644\n--- a/t/Makefile\n+++ b/t/Makefile\n@@ -14,18 +14,25 @@ SHELL_PATH_SQ = $(subst ','\\'',$(SHELL_PATH))\n T = $(wildcard t[0-9][0-9][0-9][0-9]-*.sh)\n TSVN = $(wildcard t91[0-9][0-9]-*.sh)\n \n-all: $(T) clean\n+all: pre-clean $(T) aggregate-results clean\n \n $(T):\n \t@echo \"*** $@ ***\"; GIT_CONFIG=.git/config '$(SHELL_PATH_SQ)' $@ $(GIT_TEST_OPTS)\n \n+pre-clean:\n+\t$(RM) -f test-results\n+\n clean:\n \t$(RM) -r 'trash directory'\n+\t$(RM) -f test-results\n+\n+aggregate-results:\n+\t./aggregate-results.sh\n \n # we can test NO_OPTIMIZE_COMMITS independently of LC_ALL\n full-svn-test:\n \t$(MAKE) $(TSVN) GIT_SVN_NO_OPTIMIZE_COMMITS=1 LC_ALL=C\n \t$(MAKE) $(TSVN) GIT_SVN_NO_OPTIMIZE_COMMITS=0 LC_ALL=en_US.UTF-8\n \n-.PHONY: $(T) clean\n+.PHONY: pre-clean $(T) aggregate-results clean\n .NOTPARALLEL:\n-- \n1.5.5.1.214.g82ce2.dirty\n"},{"id":"79115","messageId":"m3tzg4vvgw.fsf@localhost.localdomain","threadId":"13861","inReplyTo":"1212933875-29947-1-git-send-email-sverre@rabbelier.nl","subject":"Re: [PATCH 1/3] Modified test-lib.sh to output stats to /tmp/git-test-results","fromName":"Jakub Narebski","fromEmail":"jnareb@gmail.com","sentAt":"2008-06-08T14:42:59Z","receivedAt":"2008-06-08T14:42:59Z","isPatch":true,"sender":{"key":"jnareb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/2706?v=4"},"body":"\"Sverre Rabbelier\" writes:\n\n> Because writing to the current directory is not possible, we write to /tmp/.\n> Suggestions for a better location are welcome.\n\n> +\ttest_results_path=\"../test-results\"\n\nErrr... it looks like you forgot to update commit message.\n\nBut that aside, I quite like this series.\n-- \nJakub Narebski\nPoland\nShadeHawk on #git\n"},{"id":"79116","messageId":"bd6139dc0806080745i2ff2489bv70a1596bcc83f700@mail.gmail.com","threadId":"13861","inReplyTo":"m3tzg4vvgw.fsf@localhost.localdomain","subject":"Re: [PATCH 1/3] Modified test-lib.sh to output stats to /tmp/git-test-results","fromName":"Sverre Rabbelier","fromEmail":"srabbelier@gmail.com","sentAt":"2008-06-08T14:45:27Z","receivedAt":"2008-06-08T14:45:27Z","isPatch":true,"sender":{"key":"srabbelier@gmail.com","avatar":"https://avatars.githubusercontent.com/u/3098?v=4"},"body":"On Sun, Jun 8, 2008 at 4:42 PM, Jakub Narebski wrote:\n> \"Sverre Rabbelier\" writes:\n>\n>> Because writing to the current directory is not possible, we write to /tmp/.\n>> Suggestions for a better location are welcome.\n>\n>> + test_results_path=\"../test-results\"\n>\n> Errr... it looks like you forgot to update commit message.\n\nYeah, I intended to fix that when amending the original commit, but I\nforgot. I thought I couldn't write to the current dir, but instead I\nwas writing to /t/trash, which of course gets deleted after every\ntest. Obviously outputting to ../test-results fixed that.\n\n> But that aside, I quite like this series.\n\nThanks!\n\n-- \nCheers,\n\nSverre Rabbelier\n"},{"id":"79150","messageId":"7vabhvkbbi.fsf@gitster.siamese.dyndns.org","threadId":"13861","inReplyTo":"bd6139dc0806080745i2ff2489bv70a1596bcc83f700@mail.gmail.com","subject":"Re: [PATCH 1/3] Modified test-lib.sh to output stats to /tmp/git-test-results","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2008-06-08T18:53:37Z","receivedAt":"2008-06-08T18:53:37Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"\"Sverre Rabbelier\" writes:\n\n> On Sun, Jun 8, 2008 at 4:42 PM, Jakub Narebski wrote:\n>> \"Sverre Rabbelier\" writes:\n>>\n>>> Because writing to the current directory is not possible, we write to /tmp/.\n>>> Suggestions for a better location are welcome.\n>>\n>>> + test_results_path=\"../test-results\"\n>>\n>> Errr... it looks like you forgot to update commit message.\n>\n> Yeah, I intended to fix that when amending the original commit, but I\n> forgot. I thought I couldn't write to the current dir, but instead I\n> was writing to /t/trash, which of course gets deleted after every\n> test. Obviously outputting to ../test-results fixed that.\n\nSome tests may chdir around and when you hit test_done they may not be in\nthe directory directly under t/ anymore. I'd say that such a test is\nill-mannered, but we would want to protect ourselves against such mischief\nsomehow.\n\nAlso, this won't be an immediate issue, but we may want to lift the\n\".NOTPARALLEL\" limitation in t/Makefile sometime in the future. A handful\ntests need to be adjusted for this change, as they currently refer their\ntest vectors in t/tXXXX/ as ../tXXXX, and with such a change, we would run\neach test in \"t/trash directory/t$$\" where $$ is the pid, or something\nlike that. A single \"test-results\" file everybody races to append would\nbecome unwieldy at that point. Something to keep in mind.\n"},{"id":"79151","messageId":"bd6139dc0806081202l48953181q7d337455c85bbe2c@mail.gmail.com","threadId":"13861","inReplyTo":"7vabhvkbbi.fsf@gitster.siamese.dyndns.org","subject":"Re: [PATCH 1/3] Modified test-lib.sh to output stats to /tmp/git-test-results","fromName":"Sverre Rabbelier","fromEmail":"srabbelier@gmail.com","sentAt":"2008-06-08T19:02:47Z","receivedAt":"2008-06-08T19:02:47Z","isPatch":true,"sender":{"key":"srabbelier@gmail.com","avatar":"https://avatars.githubusercontent.com/u/3098?v=4"},"body":"On Sun, Jun 8, 2008 at 8:53 PM, Junio C Hamano wrote:\n> Some tests may chdir around and when you hit test_done they may not be in\n> the directory directly under t/ anymore. I'd say that such a test is\n> ill-mannered, but we would want to protect ourselves against such mischief\n> somehow.\n\nAh, I guess logging to /tmp/git-test-result was more reliable then.\nAny suggestions as to what is the best solution here? (We could for\nexample save $PWD at the beginning of test-lib.sh.)\n\n> Also, this won't be an immediate issue, but we may want to lift the\n> \".NOTPARALLEL\" limitation in t/Makefile sometime in the future. A handful\n> tests need to be adjusted for this change, as they currently refer their\n> test vectors in t/tXXXX/ as ../tXXXX, and with such a change, we would run\n> each test in \"t/trash directory/t$$\" where $$ is the pid, or something\n> like that. A single \"test-results\" file everybody races to append would\n> become unwieldy at that point. Something to keep in mind.\n\nAh, yes, I was going to have each test log to their own file\n(test-results-$$) and then cat the result together, but I figured that\nsince we are .NOTPARALLEL anyway it would be more efficient not to. I\nreckon that whenever we decide to make such a change it will be then\nthat we modify this script, if it is included?\n\n-- \nCheers,\n\nSverre Rabbelier\n"},{"id":"79161","messageId":"7vej77iqxb.fsf@gitster.siamese.dyndns.org","threadId":"13861","inReplyTo":"bd6139dc0806081202l48953181q7d337455c85bbe2c@mail.gmail.com","subject":"Re: [PATCH 1/3] Modified test-lib.sh to output stats to /tmp/git-test-results","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2008-06-08T20:59:28Z","receivedAt":"2008-06-08T20:59:28Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"\"Sverre Rabbelier\" writes:\n\n> Ah, I guess logging to /tmp/git-test-result was more reliable then.\n> Any suggestions as to what is the best solution here? (We could for\n> example save $PWD at the beginning of test-lib.sh.)\n\nSaving at the beginning to use in test_done, and optionally detecting \nthat cwd was changed and warn, would be sensible, I think.\n\n> Ah, yes, I was going to have each test log to their own file\n> (test-results-$$) and then cat the result together, but I figured that\n> since we are .NOTPARALLEL anyway it would be more efficient not to. I\n> reckon that whenever we decide to make such a change it will be then\n> that we modify this script, if it is included?\n\nYup. That is what I meant by \"not immediate, but something to keep in\nmind\".\n"},{"id":"79193","messageId":"1212972241-7348-1-git-send-email-srabbelier@gmail.com","threadId":"13861","inReplyTo":"7vej77iqxb.fsf@gitster.siamese.dyndns.org","subject":"[PATCH 4/4] Make test-lib warn if the directory changes.","fromName":"Sverre Rabbelier","fromEmail":"srabbelier@gmail.com","sentAt":"2008-06-09T00:44:01Z","receivedAt":"2008-06-09T00:44:01Z","isPatch":true,"sender":{"key":"srabbelier@gmail.com","avatar":"https://avatars.githubusercontent.com/u/3098?v=4"},"body":"From: Sverre Rabbelier \n\nIf a script moves out of it's directory, but thereafter\ndoes not move back, the test results would not get written\nto the proper file, as such, remember where we start at\nand then later on, write to that place.\nAlso, if it is noticed that the directory was changed,\nissue a warning.\n\nSigned-off-by: Sverre Rabbelier \n---\n t/test-lib.sh | 8 +++++++-\n 1 files changed, 7 insertions(+), 1 deletions(-)\n\ndiff --git a/t/test-lib.sh b/t/test-lib.sh\nindex 4585fde..d9f2f4e 100644\n--- a/t/test-lib.sh\n+++ b/t/test-lib.sh\n@@ -5,6 +5,7 @@\n \n # Keep the original TERM for say_color\n ORIGINAL_TERM=$TERM\n+ORIGINAL_PATH=$PWD\n \n # For repeatability, reset the environment to known value.\n LANG=C\n@@ -355,7 +356,12 @@ test_create_repo () {\n \n test_done () {\n \ttrap - exit\n-\ttest_results_path=\"../test-results\"\n+\ttest_results_path=\"$ORIGINAL_PATH/test-results\"\n+\n+ if test \"$ORIGINAL_PATH\" != \"$PWD\"\n+ then\n+ say_color error \"Script changed directory from '$ORIGINAL_PATH' to '$PWD'!\"\n+ fi\n \n \techo \"total $test_count\" >> $test_results_path\n \techo \"success $test_success\" >> $test_results_path\n-- \n1.5.6.rc2\n"}]}