{"thread":{"id":"13992","subject":"[PATCH] test-lib.sh: add --long-tests option","startedAt":"2008-06-17T01:29:02Z","lastAt":"2008-06-17T07:15:52Z","messageCount":4,"participants":["Lea Wiemann","Junio C Hamano"],"isPatch":true,"patchVersion":1,"patchTotal":null},"messages":[{"id":"80100","messageId":"1213666142-24680-1-git-send-email-LeWiemann@gmail.com","threadId":"13992","inReplyTo":null,"subject":"[PATCH] test-lib.sh: add --long-tests option","fromName":"Lea Wiemann","fromEmail":"lewiemann@gmail.com","sentAt":"2008-06-17T01:29:02Z","receivedAt":"2008-06-17T01:29:02Z","isPatch":true,"sender":{"key":"lewiemann@gmail.com","avatar":null},"body":"Add a --long-tests option to test-lib.sh, which enables tests to\nselectively run more exhaustive (longer running, potentially\nbrute-force) tests.  Such exhaustive tests would only be useful if one\nworks on the specific module that is being tested -- for a general \"cd\nt/; make\" to check whether everything is OK, such exhaustive tests\nshouldn't be run by default since the longer it takes to run the\ntests, the less often they are actually run.\n\nSigned-off-by: Lea Wiemann <LeWiemann@gmail.com>\n---\nRight now I'm using this in the Mechanize test for exhaustive link\nchecking on each page.  I've caught bugs with this (such as the\n\"[PATCH] gitweb: fix support for repository directories with spaces\"\nthat I just sent), so it's actually useful for development, but it's\nso slow that I wouldn't want it to be run unless the user requests it\n-- the link checking is really quite brute-force and doesn't cover\nmuch more than the actual test.  Actual breakages (regressions) are\nquite likely to be caught by the normal test.\n\nRunning the Mechanize test in long mode takes more than 1 minute on my\nsystem, vs 5 seconds in normal mode -- and that's just with a really\nshort draft of the test suite.  So running those long tests\nunconditionally would slow down \"cd t; make\" unncessarily, and we want\ntests to be fast so that people actually run them.\n\n\nIs the wording OK?  --long-tests and GIT_TEST_LONG were the best terms\nI could come up with off the top of my head.\n\n\n(ISTR that there's some large open source project that uses this\nstrategy (long vs. normal tests) quite extensively, but I can't recall\nwhich one; hints appreciated.  GCC uses it in its compatibility tests;\nits testsuite/README.compat reads,\n\n  Normally, only a small amount of compatibility tests is run.\n  Setting RUN_ALL_COMPAT_TESTS=1 in the environment before running the\n  testsuite enables running all compatibility tests, but might take\n  significantly longer than it takes without this variable.)\n\n-- Lea\n\n t/README      |    4 ++++\n t/test-lib.sh |    2 ++\n 2 files changed, 6 insertions(+), 0 deletions(-)\n\ndiff --git a/t/README b/t/README\nindex 70841a4..dc89263 100644\n--- a/t/README\n+++ b/t/README\n@@ -54,6 +54,10 @@ You can pass --verbose (or -v), --debug (or -d), and --immediate\n \tThis causes the test to immediately exit upon the first\n \tfailed test.\n \n+--long-tests::\n+\tThis causes additional long-running tests to be run (where\n+\tavailable), for more exhaustive testing.\n+\n \n Naming Tests\n ------------\ndiff --git a/t/test-lib.sh b/t/test-lib.sh\nindex 163167c..4cd99af 100644\n--- a/t/test-lib.sh\n+++ b/t/test-lib.sh\n@@ -80,6 +80,8 @@ do\n \t\tdebug=t; shift ;;\n \t-i|--i|--im|--imm|--imme|--immed|--immedi|--immedia|--immediat|--immediate)\n \t\timmediate=t; shift ;;\n+\t-l|--l|--lo|--lon|--long|--long-|--long-t|--long-te|--long-tes|--long-test|--long-tests)\n+\t\texport GIT_TEST_LONG=t; shift ;;\n \t-h|--h|--he|--hel|--help)\n \t\thelp=t; shift ;;\n \t-v|--v|--ve|--ver|--verb|--verbo|--verbos|--verbose)\n-- \n1.5.6.rc3.7.ged9620\n"},{"id":"80104","messageId":"7vy7547egk.fsf@gitster.siamese.dyndns.org","threadId":"13992","inReplyTo":"1213666142-24680-1-git-send-email-LeWiemann@gmail.com","subject":"Re: [PATCH] test-lib.sh: add --long-tests option","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2008-06-17T06:39:07Z","receivedAt":"2008-06-17T06:39:07Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Lea Wiemann <lewiemann@gmail.com> writes:\n\n> diff --git a/t/test-lib.sh b/t/test-lib.sh\n> index 163167c..4cd99af 100644\n> --- a/t/test-lib.sh\n> +++ b/t/test-lib.sh\n> @@ -80,6 +80,8 @@ do\n>  \t\tdebug=t; shift ;;\n>  \t-i|--i|--im|--imm|--imme|--immed|--immedi|--immedia|--immediat|--immediate)\n>  \t\timmediate=t; shift ;;\n> +\t-l|--l|--lo|--lon|--long|--long-|--long-t|--long-te|--long-tes|--long-test|--long-tests)\n> +\t\texport GIT_TEST_LONG=t; shift ;;\n>  \t-h|--h|--he|--hel|--help)\n>  \t\thelp=t; shift ;;\n>  \t-v|--v|--ve|--ver|--verb|--verbo|--verbos|--verbose)\n\nHmm.\n\nI am guessing that the reason why you do not unset GIT_TEST_LONG upfront\nin the script is because the user can do:\n\n\t$ cd t\n        $ GIT_TEST_LONG=t; export GIT_TEST_LONG\n        $ ./t9999-this-test.sh\n        $ ./t9999-that-test.sh\n\nor even:\n\n\t$ GIT_TEST_LONG=t make test\n\nwithout having to say --long-tests from the command line that way.  If\nthat is the case, however, I wonder if this --long-tests option is even\nnecessary.\n\nPlease do not get me wrong --- I am _not_ suggesting to unset (or set to\nempty) GIT_TEST_LONG at the beginning of test-lib before command line\nparsing.  I do want the ability to run all the expensive tests with a\nsingle command line retained, as I expect it would be useful before a\nmajor release.\n"},{"id":"80110","messageId":"48576345.7000809@gmail.com","threadId":"13992","inReplyTo":"7vy7547egk.fsf@gitster.siamese.dyndns.org","subject":"Re: [PATCH] test-lib.sh: add --long-tests option","fromName":"Lea Wiemann","fromEmail":"lewiemann@gmail.com","sentAt":"2008-06-17T07:09:57Z","receivedAt":"2008-06-17T07:09:57Z","isPatch":true,"sender":{"key":"lewiemann@gmail.com","avatar":null},"body":"Junio C Hamano wrote:\n> I am guessing that the reason why you do not unset GIT_TEST_LONG upfront\n> in the script is because the user can do:\n> \n>         $ GIT_TEST_LONG=t; export GIT_TEST_LONG\n>         $ ./t9999-this-test.sh\n>         $ ./t9999-that-test.sh\n> \n> or even:\n> \n> \t$ GIT_TEST_LONG=t make test\n\nYes, that's the idea.\n\n> If that is the case, however, I wonder if this --long-tests option is even\n> necessary.\n\nIt's very convenient for development, where you go back and forth \nbetween \"./t9503-gitweb-Mechanize.sh -v -l\" and \n\"./t9503-gitweb-Mechanize.sh -v\".  All it takes is to add or remove the \n\"-l\" at each invocation.  So I think the switch is actually quite useful.\n\n-- Lea\n"},{"id":"80111","messageId":"7vprqg7crb.fsf@gitster.siamese.dyndns.org","threadId":"13992","inReplyTo":"48576345.7000809@gmail.com","subject":"Re: [PATCH] test-lib.sh: add --long-tests option","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2008-06-17T07:15:52Z","receivedAt":"2008-06-17T07:15:52Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Lea Wiemann <lewiemann@gmail.com> writes:\n\n> Junio C Hamano wrote:\n>> I am guessing that the reason why you do not unset GIT_TEST_LONG upfront\n>> in the script is because the user can do:\n>>\n>>         $ GIT_TEST_LONG=t; export GIT_TEST_LONG\n>>         $ ./t9999-this-test.sh\n>>         $ ./t9999-that-test.sh\n>>\n>> or even:\n>>\n>> \t$ GIT_TEST_LONG=t make test\n>\n> Yes, that's the idea.\n>\n>> If that is the case, however, I wonder if this --long-tests option is even\n>> necessary.\n>\n> It's very convenient for development, where you go back and forth\n> between \"./t9503-gitweb-Mechanize.sh -v -l\" and\n> \"./t9503-gitweb-Mechanize.sh -v\".  All it takes is to add or remove\n> the \"-l\" at each invocation.  So I think the switch is actually quite\n> useful.\n\nHeh, one-shot export \"GIT_TEST_LONG_t ./t9999-this-test.sh\" would equally\nbe swift, but Ok.\n"}]}