{"thread":{"id":"24254","subject":"[PATCH 1/8] t/README: Tests are all +x, ./test, not sh ./test","startedAt":"2010-07-01T15:09:56Z","lastAt":"2010-07-01T17:16:36Z","messageCount":14,"participants":["Ævar Arnfjörð Bjarmason","Jeff King","Junio C Hamano"],"isPatch":true,"patchVersion":1,"patchTotal":8},"messages":[{"id":"144600","messageId":"1277997004-29504-1-git-send-email-avarab@gmail.com","threadId":"24254","inReplyTo":null,"subject":"[PATCH 0/8] Improvements for t/README","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2010-07-01T15:09:56Z","receivedAt":"2010-07-01T15:09:56Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"Here are the the t/README improvements I promised Junio in\n<AANLkTilYoGtTIr9lIQD4sx4PRjlWYjIwSDMquREzv6eq@mail.gmail.com>, and\nmore. I took the opportunity while I was at it to document some\nnon-TAP things in the test suite like the prereq functions and\ntest_external_*.\n\nÆvar Arnfjörð Bjarmason (8):\n  t/README: Tests are all +x, ./test, not sh ./test\n  t/README: The trash is in 't/trash directory.$name'\n  t/README: Typo: paralell -> parallel\n  t/README: Document the prereq functions, and 3-arg test_*\n  t/README: Document test_external*\n  t/README: Document test_expect_code\n  t/README: Add a section about skipping tests\n  t/README: Document the dangers of printing \"ok\" / \"not ok\"\n\n t/README |  100 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++-----\n 1 files changed, 92 insertions(+), 8 deletions(-)\n"},{"id":"144595","messageId":"1277997004-29504-2-git-send-email-avarab@gmail.com","threadId":"24254","inReplyTo":"1277997004-29504-1-git-send-email-avarab@gmail.com","subject":"[PATCH 1/8] t/README: Tests are all +x, ./test, not sh ./test","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2010-07-01T15:09:57Z","receivedAt":"2010-07-01T15:09:57Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"They all run with a #!/bin/sh shebang, but it's probably better to use\nthe ./ form anyway.\n\nSigned-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>\n---\n t/README |    2 +-\n 1 files changed, 1 insertions(+), 1 deletions(-)\n\ndiff --git a/t/README b/t/README\nindex aa4ed28..b378ab1 100644\n--- a/t/README\n+++ b/t/README\n@@ -52,7 +52,7 @@ prove and other harnesses come with a lot of useful options. The\n \n You can also run each test individually from command line, like this:\n \n-    $ sh ./t3010-ls-files-killed-modified.sh\n+    $ ./t3010-ls-files-killed-modified.sh\n     ok 1 - git update-index --add to add various paths.\n     ok 2 - git ls-files -k to show killed files.\n     ok 3 - validate git ls-files -k output.\n-- \n1.7.1.251.g92a7\n"},{"id":"144601","messageId":"1277997004-29504-3-git-send-email-avarab@gmail.com","threadId":"24254","inReplyTo":"1277997004-29504-1-git-send-email-avarab@gmail.com","subject":"[PATCH 2/8] t/README: The trash is in 't/trash directory.$name'","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2010-07-01T15:09:58Z","receivedAt":"2010-07-01T15:09:58Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>\n---\n t/README |    7 ++++---\n 1 files changed, 4 insertions(+), 3 deletions(-)\n\ndiff --git a/t/README b/t/README\nindex b378ab1..fd61ec7 100644\n--- a/t/README\n+++ b/t/README\n@@ -221,9 +221,10 @@ This test harness library does the following things:\n  - If the script is invoked with command line argument --help\n    (or -h), it shows the test_description and exits.\n \n- - Creates an empty test directory with an empty .git/objects\n-   database and chdir(2) into it.  This directory is 't/trash directory'\n-   if you must know, but I do not think you care.\n+ - Creates an empty test directory with an empty .git/objects database\n+   and chdir(2) into it.  This directory is 't/trash\n+   directory.$test_name_without_dotsh' if you must know, but I do not\n+   think you care.\n \n  - Defines standard test helper functions for your scripts to\n    use.  These functions are designed to make all scripts behave\n-- \n1.7.1.251.g92a7\n"},{"id":"144599","messageId":"1277997004-29504-4-git-send-email-avarab@gmail.com","threadId":"24254","inReplyTo":"1277997004-29504-1-git-send-email-avarab@gmail.com","subject":"[PATCH 3/8] t/README: Typo: paralell -> parallel","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2010-07-01T15:09:59Z","receivedAt":"2010-07-01T15:09:59Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>\n---\n t/README |    2 +-\n 1 files changed, 1 insertions(+), 1 deletions(-)\n\ndiff --git a/t/README b/t/README\nindex fd61ec7..0998bfb 100644\n--- a/t/README\n+++ b/t/README\n@@ -33,7 +33,7 @@ the tests.\n     ok 3 - plain bare\n \n Since the tests all output TAP (see http://testanything.org) they can\n-be run with any TAP harness. Here's an example of paralell testing\n+be run with any TAP harness. Here's an example of parallel testing\n powered by a recent version of prove(1):\n \n     $ prove --timer --jobs 15 ./t[0-9]*.sh\n-- \n1.7.1.251.g92a7\n"},{"id":"144596","messageId":"1277997004-29504-5-git-send-email-avarab@gmail.com","threadId":"24254","inReplyTo":"1277997004-29504-1-git-send-email-avarab@gmail.com","subject":"[PATCH 4/8] t/README: Document the prereq functions, and 3-arg test_*","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2010-07-01T15:10:00Z","receivedAt":"2010-07-01T15:10:00Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"There was no documentation for the test_set_prereq and\ntest_have_prereq functions, or the three-arg form of\ntest_expect_success and test_expect_failure.\n\nSigned-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>\n---\n t/README |   37 ++++++++++++++++++++++++++++++++++---\n 1 files changed, 34 insertions(+), 3 deletions(-)\n\ndiff --git a/t/README b/t/README\nindex 0998bfb..67f8f30 100644\n--- a/t/README\n+++ b/t/README\n@@ -246,9 +246,9 @@ Test harness library\n There are a handful helper functions defined in the test harness\n library for your script to use.\n \n- - test_expect_success <message> <script>\n+ - test_expect_success [<prereq>] <message> <script>\n \n-   This takes two strings as parameter, and evaluates the\n+   Usually takes two strings as parameter, and evaluates the\n    <script>.  If it yields success, test is considered\n    successful.  <message> should state what it is testing.\n \n@@ -257,8 +257,15 @@ library for your script to use.\n \ttest_expect_success \\\n \t    'git-write-tree should be able to write an empty tree.' \\\n \t    'tree=$(git-write-tree)'\n+        \n+   If you supply three parameters the first will be taken to be a\n+   prerequisite, see the test_set_prereq and test_have_prereq\n+   documentation below:\n \n- - test_expect_failure <message> <script>\n+\ttest_expect_success TTY 'git --paginate rev-list uses a pager' \\\n+\t    ' ... '\n+\n+ - test_expect_failure [<prereq>] <message> <script>\n \n    This is NOT the opposite of test_expect_success, but is used\n    to mark a test that demonstrates a known breakage.  Unlike\n@@ -266,6 +273,9 @@ library for your script to use.\n    success and \"FAIL\" on failure, this will say \"FIXED\" on\n    success and \"still broken\" on failure.  Failures from these\n    tests won't cause -i (immediate) to stop.\n+   \n+   Like test_expect_success this function can optionally use a three\n+   argument invocation with a prerequisite as the first argument.\n \n  - test_debug <script>\n \n@@ -299,6 +309,27 @@ library for your script to use.\n    Merges the given rev using the given message.  Like test_commit,\n    creates a tag and calls test_tick before committing.\n \n+ - test_set_prereq SOME_PREREQ\n+ \n+   Set a test prerequisite to be used later with test_have_prereq. The\n+   test-lib will set some prerequisites for you, e.g. PERL and PYTHON\n+   which are derived from ./GIT-BUILD-OPTIONS (grep test_set_prereq\n+   test-lib.sh for more). Others you can set yourself and use later\n+   with either test_have_prereq directly, or the three argument\n+   invocation of test_expect_success and test_expect_failure.\n+   \n+ - test_have_prereq SOME PREREQ\n+\n+   Check if we have a prerequisite previously set with\n+   test_set_prereq. The most common use of this directly is to skip\n+   all the tests if we don't have some essential prerequisite:\n+ \n+\tif ! test_have_prereq PERL\n+\tthen\n+\t    skip_all='skipping perl interface tests, perl not available'\n+\t    test_done\n+\tfi\n+\n Tips for Writing Tests\n ----------------------\n \n-- \n1.7.1.251.g92a7\n"},{"id":"144602","messageId":"1277997004-29504-6-git-send-email-avarab@gmail.com","threadId":"24254","inReplyTo":"1277997004-29504-1-git-send-email-avarab@gmail.com","subject":"[PATCH 5/8] t/README: Document test_external*","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2010-07-01T15:10:01Z","receivedAt":"2010-07-01T15:10:01Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"There was do documentation for the test_external_without_stderr and\ntest_external functions.\n\nSigned-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>\n---\n t/README |   27 +++++++++++++++++++++++++++\n 1 files changed, 27 insertions(+), 0 deletions(-)\n\ndiff --git a/t/README b/t/README\nindex 67f8f30..43d6d4d 100644\n--- a/t/README\n+++ b/t/README\n@@ -329,6 +329,33 @@ library for your script to use.\n \t    skip_all='skipping perl interface tests, perl not available'\n \t    test_done\n \tfi\n+    \n+ - test_external [<prereq>] <message> <external> <script>\n+ \n+   Execute a <script> with an <external> interpreter (like perl). This\n+   was added for tests like t9700-perl-git.sh which do most of their\n+   work in an external test script.\n+   \n+\ttest_external_without_stderr \\\n+\t    'Perl API' \\\n+\t    \"$PERL_PATH\" \"$TEST_DIRECTORY\"/t9700/test.pl\n+\n+   If the test is outputting its own TAP you should set the\n+   test_external_has_tap variable somewhere before calling the first\n+   test_external* function. See t9700-perl-git.sh for an example.\n+\n+\t# The external test will outputs its own plan\n+\ttest_external_has_tap=1\n+    \n+ - test_external_without_stderr [<prereq>] <message> <external> <script>\n+ \n+   Like test_external but fail if there's any output on stderr,\n+   instead of checking the exit code.\n+ \n+\ttest_external_without_stderr \\\n+\t    'Perl API' \\\n+\t    \"$PERL_PATH\" \"$TEST_DIRECTORY\"/t9700/test.pl\n+\n \n Tips for Writing Tests\n ----------------------\n-- \n1.7.1.251.g92a7\n"},{"id":"144598","messageId":"1277997004-29504-7-git-send-email-avarab@gmail.com","threadId":"24254","inReplyTo":"1277997004-29504-1-git-send-email-avarab@gmail.com","subject":"[PATCH 6/8] t/README: Document test_expect_code","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2010-07-01T15:10:02Z","receivedAt":"2010-07-01T15:10:02Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"test_expect_code (which was introduced in d3bfdb75) never had any\ndocumentation.\n\nSigned-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>\n---\n t/README |    7 +++++++\n 1 files changed, 7 insertions(+), 0 deletions(-)\n\ndiff --git a/t/README b/t/README\nindex 43d6d4d..6232567 100644\n--- a/t/README\n+++ b/t/README\n@@ -276,6 +276,13 @@ library for your script to use.\n    \n    Like test_expect_success this function can optionally use a three\n    argument invocation with a prerequisite as the first argument.\n+   \n+ - test_expect_code [<prereq>] <code> <message> <script>\n+ \n+   Analogous to test_expect_success, but pass the test if it exits\n+   with a given exit <code>\n+ \n+ test_expect_code 1 'Merge with d/f conflicts' 'git merge \"merge msg\" B master'\n \n  - test_debug <script>\n \n-- \n1.7.1.251.g92a7\n"},{"id":"144597","messageId":"1277997004-29504-8-git-send-email-avarab@gmail.com","threadId":"24254","inReplyTo":"1277997004-29504-1-git-send-email-avarab@gmail.com","subject":"[PATCH 7/8] t/README: Add a section about skipping tests","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2010-07-01T15:10:03Z","receivedAt":"2010-07-01T15:10:03Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>\n---\n t/README |   12 ++++++++++++\n 1 files changed, 12 insertions(+), 0 deletions(-)\n\ndiff --git a/t/README b/t/README\nindex 6232567..42a4c25 100644\n--- a/t/README\n+++ b/t/README\n@@ -231,6 +231,18 @@ This test harness library does the following things:\n    consistently when command line arguments --verbose (or -v),\n    --debug (or -d), and --immediate (or -i) is given.\n \n+Skipping tests\n+--------------\n+\n+If you need to skip all the remaining tests you should set skip_all\n+and immediately call test_done. The string you give to skip_all will\n+be used as an explanation for why the test was skipped. for instance:\n+\n+\tif ! test_have_prereq PERL\n+\tthen\n+\t    skip_all='skipping perl interface tests, perl not available'\n+\t    test_done\n+\tfi\n \n End with test_done\n ------------------\n-- \n1.7.1.251.g92a7\n"},{"id":"144603","messageId":"1277997004-29504-9-git-send-email-avarab@gmail.com","threadId":"24254","inReplyTo":"1277997004-29504-1-git-send-email-avarab@gmail.com","subject":"[PATCH 8/8] t/README: Document the dangers of printing \"ok\" / \"not ok\"","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2010-07-01T15:10:04Z","receivedAt":"2010-07-01T15:10:04Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"A TAP harness will get confused if you print \"ok\" or \"not ok\" at the\nbeginning of a line. This had to be fixed in\n335f87871fe5aa6b3fd55b2b4e80f16fe9681483 when TAP support was\nintroduced.\n\nSigned-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>\n---\n t/README |    6 ++++++\n 1 files changed, 6 insertions(+), 0 deletions(-)\n\ndiff --git a/t/README b/t/README\nindex 42a4c25..29f795e 100644\n--- a/t/README\n+++ b/t/README\n@@ -50,6 +50,12 @@ prove and other harnesses come with a lot of useful options. The\n     # Repeat until no more failures\n     $ prove -j 15 --state=failed,save ./t[0-9]*.sh\n \n+The TAP support is completely incidental. A TAP harness is just a\n+program that interprets the \"ok\"/\"not ok\" (and some other strings) in\n+a special way. The only limitation this imposes is that you shouldn't\n+echo \"ok\" or \"not ok\" by yourself at the beginning of a line, that'll\n+confuse the TAP harness.\n+\n You can also run each test individually from command line, like this:\n \n     $ ./t3010-ls-files-killed-modified.sh\n-- \n1.7.1.251.g92a7\n"},{"id":"144605","messageId":"20100701154211.GA15542@sigill.intra.peff.net","threadId":"24254","inReplyTo":"1277997004-29504-2-git-send-email-avarab@gmail.com","subject":"Re: [PATCH 1/8] t/README: Tests are all +x, ./test, not sh ./test","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2010-07-01T15:42:12Z","receivedAt":"2010-07-01T15:42:12Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Thu, Jul 01, 2010 at 03:09:57PM +0000, Ævar Arnfjörð Bjarmason wrote:\n\n> They all run with a #!/bin/sh shebang, but it's probably better to use\n> the ./ form anyway.\n\nIf we want to be nitpicky, they are actually run by \"make test\" with\n$SHELL_PATH. We don't re-write the shebang lines in the test scripts (as\nwe do with the actual git-* scripts), so running \"./t-whatever\" won't\nwork if you have a broken shell (e.g., Solaris). You have to do \"bash\n./t-whatever\".\n\n-Peff\n"},{"id":"144606","messageId":"20100701154429.GB15542@sigill.intra.peff.net","threadId":"24254","inReplyTo":"1277997004-29504-3-git-send-email-avarab@gmail.com","subject":"Re: [PATCH 2/8] t/README: The trash is in 't/trash directory.$name'","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2010-07-01T15:44:29Z","receivedAt":"2010-07-01T15:44:29Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Thu, Jul 01, 2010 at 03:09:58PM +0000, Ævar Arnfjörð Bjarmason wrote:\n\n> diff --git a/t/README b/t/README\n> index b378ab1..fd61ec7 100644\n> --- a/t/README\n> +++ b/t/README\n> @@ -221,9 +221,10 @@ This test harness library does the following things:\n>   - If the script is invoked with command line argument --help\n>     (or -h), it shows the test_description and exits.\n>  \n> - - Creates an empty test directory with an empty .git/objects\n> -   database and chdir(2) into it.  This directory is 't/trash directory'\n> -   if you must know, but I do not think you care.\n> + - Creates an empty test directory with an empty .git/objects database\n> +   and chdir(2) into it.  This directory is 't/trash\n> +   directory.$test_name_without_dotsh' if you must know, but I do not\n> +   think you care.\n\nYou might want to mention the impact of \"--root\" here. I.e., it is\nactually \"$root${root:+/}trash directory.$test_name_without_dotsh\", and\nit is in t/ only if $root is not an absolute path.\n\n-Peff\n"},{"id":"144610","messageId":"7vwrtf2m5r.fsf@alter.siamese.dyndns.org","threadId":"24254","inReplyTo":"1277997004-29504-5-git-send-email-avarab@gmail.com","subject":"Re: [PATCH 4/8] t/README: Document the prereq functions, and 3-arg test_*","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2010-07-01T16:37:36Z","receivedAt":"2010-07-01T16:37:36Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Thanks.  We should have done this one long time ago.\n"},{"id":"144611","messageId":"7vsk432m3h.fsf@alter.siamese.dyndns.org","threadId":"24254","inReplyTo":"1277997004-29504-6-git-send-email-avarab@gmail.com","subject":"Re: [PATCH 5/8] t/README: Document test_external*","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2010-07-01T16:38:58Z","receivedAt":"2010-07-01T16:38:58Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Thanks.  We should have done this one long time ago, too.\n"},{"id":"144614","messageId":"7vhbkj2kcr.fsf@alter.siamese.dyndns.org","threadId":"24254","inReplyTo":"1277997004-29504-9-git-send-email-avarab@gmail.com","subject":"Re: [PATCH 8/8] t/README: Document the dangers of printing \"ok\" / \"not ok\"","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2010-07-01T17:16:36Z","receivedAt":"2010-07-01T17:16:36Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Ævar Arnfjörð Bjarmason  <avarab@gmail.com> writes:\n\n> index 42a4c25..29f795e 100644\n> --- a/t/README\n> +++ b/t/README\n> @@ -50,6 +50,12 @@ prove and other harnesses come with a lot of useful options. The\n>      # Repeat until no more failures\n>      $ prove -j 15 --state=failed,save ./t[0-9]*.sh\n>  \n> +The TAP support is completely incidental. A TAP harness is just a\n> +program that interprets the \"ok\"/\"not ok\" (and some other strings) in\n> +a special way. The only limitation this imposes is that you shouldn't\n> +echo \"ok\" or \"not ok\" by yourself at the beginning of a line, that'll\n> +confuse the TAP harness.\n> +\n\nI think the above is not suitable for the target audience of the section\n(\"Running Tests\"); they are not _writing_ tests.\n\nThis information (or an enhanced version of it) should come between\n\"Source 'test-lib.sh'\" that talks about what the harness gives, and\n\"Skipping tests\".  Perhaps name that to \"Test your stuff\".  An example...\n\n    Test your stuff\n    ---------------\n\n    After sourcing test-lib.sh, you test the parts of the system using the\n    test harness.  Your tests will be done in the <script> part of\n    test_expect_success helper function, and the exit status of the last\n    command in that <script> signals the success/failure of a test.\n\n    You are not supposed to \"exit\" from the <script> part; the harness\n    will catch this as a programming error of the test.  Use test_done\n    instead, if you need to stop the tests early (see \"Skipping tests\"\n    below).\n\n    Inside <script> part, the standard output and standard error streams\n    are discarded, and the test harness only reports \"ok\" or \"not ok\" to\n    the end user running the tests.  Under --verbose, they are shown to\n    help debugging the tests.\n\n    The output from the test suite is designed to be TAP compliant (see\n    http://..... to learn more about TAP), to allow programs by other\n    people to parse our output.  This further places restriction on what\n    your tests can say to the standard output and standard error.  The TAP\n    readers expect a line that begins with either \"ok\" and \"not ok\" to\n    signal a test passed or failed (and our harness already produces such\n    lines), so your script shouldn't emit such lines to their output.\n\nI suspect the last paragraph needs a bit more work.  For example, don't\nyou have some restriction not to say things like \"1..43\"???\n"}]}