git/list[1] front-page[2] threads[3] people[4] search[5] about
 

Re: [PATCH 1/2] t0303: set reason for skipping tests

From
Zbigniew Jędrzejewski-Szmek <zbyszek@in.waw.pl>
Date
Mar 12, 2012, 21:07 UTC
Message-ID
<4F5E65AE.8050401@in.waw.pl>
In-Reply-To
<20120312123031.GA14456@sigill.intra.peff.net>
On 03/12/2012 01:30 PM, Jeff King wrote:
Show 11 quoted lines
> On Mon, Mar 12, 2012 at 01:05:06PM +0100, Zbigniew Jędrzejewski-Szmek wrote:
>
>> t0300-credential-helpers.sh runs two sets of tests. Each set is
>> controlled by an environment variable and is skipped if the variable
>> is not defined. If both sets are skipped, prove will say:
>>    ./t0303-credential-external.sh .. skipped: (no reason given)
>> which isn't very nice.
>>
>> Use skip_all="..." to set the reason when both sets are skipped.
>
> Sounds reasonable. A few nits:

OK, it seems this might be more complicated than I expected. I admit that I didn't test this (apart from failing without the variables defined) and assumed that it more or less works already.

I think that the tests are not very robust:
     ln -s /bin/true ~/bin/git-credential-fooooooo
     GIT_TEST_CREDENTIAL_HELPER=fooooooo\
       GIT_TEST_CREDENTIAL_HELPER_TIMEOUT=fooooooo\
       ./t0303-credential-external.sh

gives me: ok 1 - helper (fooooooo) has no existing data ok 2 - helper (fooooooo) stores password not ok - 3 helper (fooooooo) can retrieve password ok 4 - helper (fooooooo) requires matching protocol ok 5 - helper (fooooooo) requires matching host ok 6 - helper (fooooooo) requires matching username ok 7 - helper (fooooooo) requires matching path ok 8 - helper (fooooooo) can forget host not ok - 9 helper (fooooooo) can store multiple users ok 10 - helper (fooooooo) can forget user not ok - 11 helper (fooooooo) remembers other user ok 12 - helper (fooooooo) times out # failed 3 among 12 test(s) 1..12

I guess that the fact that #1 succeeds reflects reality, but e.g. 4-7 and 12 probably should fail.

Show 11 quoted lines
>>   if test -z "$GIT_TEST_CREDENTIAL_HELPER"; then
>> -	say "# skipping external helper tests (set GIT_TEST_CREDENTIAL_HELPER)"
>> +	say "# skipping external helper tests (GIT_TEST_CREDENTIAL_HELPER not set)"
>>   else
>> [...]
>>   if test -z "$GIT_TEST_CREDENTIAL_HELPER_TIMEOUT"; then
>> -	say "# skipping external helper timeout tests"
>> +	say "# skipping external helper timeout tests (GIT_TEST_CREDENTIAL_HELPER_TIMEOUT not set)"
>
> These don't affect prove at all, do they? I'm OK with the changes, but I
> was confused to see them after reading the commit message.

Right, this was just for symmetry when running test directly. Forgot to mention this in the commit message.

> Should they actually say "# SKIP ..." to tell prove what's going on? I
> don't know very much about TAP.

# SKIP is used when skipping individual tests (IIUC), but when we skip a group of tests, we simply jump over them and this message is purely informative output that is not interpreted by the harness.

Show 29 quoted lines
>> +if test -z "$GIT_TEST_CREDENTIAL_HELPER" \
>> +    -o -z "$GIT_TEST_CREDENTIAL_HELPER_TIMEOUT"; then
>> +    skip_all="used to test external credential helpers"
>> +fi
>
> Actually, I think it is not OK to run t0303 with HELPER_TIMEOUT set, but
> HELPER not set. The "helper_test_clean" bits will fail badly. The
> documentation given in the commit message is actually wrong (I added the
> clean bits to the patch later, and failed to realize the dependency or
> update the commit message).
>
> Also, our usual idiom is to check the prerequisites at the top of the
> script and bail immediately.
>
> So maybe the whole script should be restructured as:
>
>    if test -z "$GIT_TEST_CREDENTIAL_HELPER"; then
>            skip_all="GIT_TEST_CREDENTIAL_HELPER not set"
>            test_done
>    fi
>
>    pre_test
>    helper_test "$GIT_TEST_CREDENTIAL_HELPER"
>    if test -z "$GIT_TEST_CREDENTIAL_HELPER_TIMEOUT"; then
>            say "# skipping timeout tests (GIT_TEST_CREDENTIAL_HELPER_TIMEOUT not set)"
>    else
>            helper_test_timeout "$GIT_TEST_CREDENTIAL_HELPER_TIMEOUT"
>    fi
>    post_test

Yeah, this seems to be the right approach. I'll repost with a proper commit message once my doubts about the tests are cleared up.

- Zbyszek

Previous: Jeff KingNext: Jeff King
Message 4 of 23 in “prettify t0303-credential-helpers.sh”
  1. 0/2 prettify t0303-credential-helpers.shZbigniew Jędrzejewski-Szmek, Mar 12, 2012
  2. 1/2 t0303: set reason for skipping testsZbigniew Jędrzejewski-Szmek, Mar 12, 2012
  3. Jeff KingMar 12, 2012
  4. Zbigniew Jędrzejewski-SzmekMar 12, 2012
  5. Jeff KingMar 13, 2012
  6. Zbigniew Jędrzejewski-SzmekMar 14, 2012
  7. 1/2 t0303: immediately bail out w/o GIT_TEST_CREDENTIAL_HELPERZbigniew Jędrzejewski-Szmek, Mar 14, 2012
  8. 2/2 t0303: resurrect commit message as test documentationZbigniew Jędrzejewski-Szmek, Mar 14, 2012
  9. Junio C HamanoMar 14, 2012
  10. Jeff KingMar 15, 2012
  11. Junio C HamanoMar 15, 2012
  12. Zbigniew Jędrzejewski-SzmekMar 15, 2012
  13. 1/2 t0303: immediately bail out w/o GIT_TEST_CREDENTIAL_HELPERZbigniew Jędrzejewski-Szmek, Mar 15, 2012
  14. 2/2 t0303: resurrect commit message as test documentationZbigniew Jędrzejewski-Szmek, Mar 15, 2012
  15. Junio C HamanoMar 15, 2012
  16. Jeff KingMar 15, 2012
  17. Jeff KingMar 15, 2012
  18. Junio C HamanoMar 15, 2012
  19. Jeff KingMar 15, 2012
  20. 2/2 t0303: resurrect commit message as test documentationZbigniew Jędrzejewski-Szmek, Mar 12, 2012
  21. Jeff KingMar 12, 2012
  22. Jonathan NiederMar 12, 2012
  23. Jeff KingMar 13, 2012

Read the whole thread, see it on lore, or plain text.

$ cat FOOTERMessages come from the public archive at lore.kernel.org/git, fetched every hour. The front page is chosen and written each morning by an AI editor and can be wrong; the threads themselves are the record. About and API. For agents: an MCP server at https://gitlist.dev/mcp, and any thread, story or person page as Markdown by adding .md to its URL (or sending Accept: text/markdown). Details in /llms.txt.