make test

Hi folks

I'm trying to understand why certain tests in 'make test' fail. Here's the first one

$ ../git --version git version 1.8.0.rc2.5.g6b89306 $ GIT_TEST_CMP_USE_COPIED_CONTEXT=true ./t0000-basic.sh # our diff doesn't understand -u ok 1 - .git/objects should be empty after git init in an empty repo ... ok 3 - success is reported like this not ok 4 - pretend we have a known breakage # TODO known breakage

    This is expected, right?

ok 5 - pretend we have fixed a known breakage (run in sub test-lib) ... ok 11 - tests clean up after themselves

    the next is not though? Why might it be failing, where to check?

not ok - 12 tests clean up even on failures # # mkdir failing-cleanup && # ( # cd failing-cleanup && # # cat >failing-cleanup.sh <<-EOF && # #!/bin/sh # # test_description='Failing tests with cleanup commands' # # # Point to the t/test-lib.sh, which isn't in ../ as usual # TEST_DIRECTORY="/home/jojo/git/git/t" # . "$TEST_DIRECTORY"/test-lib.sh # # test_expect_success 'tests clean up even after a failure' ' # touch clean-after-failure && # test_when_finished rm clean-after-failure && # (exit 1) # ' # test_expect_success 'failure to clean up causes the test to fail' ' # test_when_finished "(exit 2)" # ' # test_done # # EOF # # chmod +x failing-cleanup.sh && # test_must_fail ./failing-cleanup.sh >out 2>err && # ! test -s err && # ! test -f "trash directory.failing-cleanup/clean-after-failure" && # sed -e 's/Z$//' -e 's/^> //' >expect <<-\EOF && # > not ok - 1 tests clean up even after a failure # > # Z # > # touch clean-after-failure && # > # test_when_finished rm clean-after-failure && # > # (exit 1) # > # Z # > not ok - 2 failure to clean up causes the test to fail # > # Z # > # test_when_finished "(exit 2)" # > # Z # > # failed 2 among 2 test(s) # > 1..2 # EOF # test_cmp expect out # ) # ok 13 - git update-index without --add should fail adding ... ok 47 - very long name in the index handled sanely # still have 1 known breakage(s) # failed 1 among remaining 46 test(s) 1..47

Bye, Jojo 

Am 10/15/2012 12:36, schrieb Joachim Schmitz:

Right.

First thing:

  ./t0000-basic.sh -v -i

and if that does not give sufficient clues,

  $SHELL_PATH -x ./t0000-basic.sh -v -i

(Beware, though: in some cases, the latter gives additional failures, in particular, when the stderr of a command is checked for with test_cmp instead of grep because the 'actual' results contain the shell command logs, which are not in the 'expected' results.)

-- Hannes,

OK, I should have mentioned that I did look with -v : not ok - 12 tests clean up even on failures # # mkdir failing-cleanup && # ( # cd failing-cleanup && # # cat >failing-cleanup.sh <<-EOF && # #!/bin/sh # # test_description='Failing tests with cleanup commands' # # # Point to the t/test-lib.sh, which isn't in ../ as usual # TEST_DIRECTORY="/home/jojo/git/git/t" # . "$TEST_DIRECTORY"/test-lib.sh # # test_expect_success 'tests clean up even after a failure' ' # touch clean-after-failure && # test_when_finished rm clean-after-failure && # (exit 1) # ' # test_expect_success 'failure to clean up causes the test to fail' ' # test_when_finished "(exit 2)" # ' # test_done # # EOF # # chmod +x failing-cleanup.sh && # test_must_fail ./failing-cleanup.sh >out 2>err && # ! test -s err && # ! test -f "trash directory.failing-cleanup/clean-after-failure" && # sed -e 's/Z$//' -e 's/^> //' >expect <<-\EOF && # > not ok - 1 tests clean up even after a failure # > # Z # > # touch clean-after-failure && # > # test_when_finished rm clean-after-failure && # > # (exit 1) # > # Z # > not ok - 2 failure to clean up causes the test to fail # > # Z # > # test_when_finished "(exit 2)" # > # Z # > # failed 2 among 2 test(s) # > 1..2 # EOF # test_cmp expect out # ) #

not ok - 12 tests clean up even on failures
#
#               mkdir failing-cleanup &&
#               (
#               cd failing-cleanup &&
#
#               cat >failing-cleanup.sh <<-EOF &&
#               #!/bin/sh
#
#               test_description='Failing tests with cleanup commands'
#
#               # Point to the t/test-lib.sh, which isn't in ../ as usual
#               TEST_DIRECTORY="/home/jojo/git/git/t"
#               . "$TEST_DIRECTORY"/test-lib.sh
#
#               test_expect_success 'tests clean up even after a failure' '
#                       touch clean-after-failure &&
#                       test_when_finished rm clean-after-failure &&
#                       (exit 1)
#               '
#               test_expect_success 'failure to clean up causes the test to fail' '
#                       test_when_finished "(exit 2)"
#               '
#               test_done
#
#               EOF
#
#               chmod +x failing-cleanup.sh &&
#               test_must_fail ./failing-cleanup.sh >out 2>err &&
#               ! test -s err &&
#               ! test -f "trash directory.failing-cleanup/clean-after-failure" &&
#               sed -e 's/Z$//' -e 's/^> //' >expect <<-\EOF &&
#               > not ok - 1 tests clean up even after a failure
#               > #     Z
#               > #     touch clean-after-failure &&
#               > #     test_when_finished rm clean-after-failure &&
#               > #     (exit 1)
#               > #     Z
#               > not ok - 2 failure to clean up causes the test to fail
#               > #     Z
#               > #     test_when_finished "(exit 2)"
#               > #     Z
#               > # failed 2 among 2 test(s)
#               > 1..2
#               EOF
#               test_cmp expect out
#               )
#
+ die

Looks identical, except for the "die" at the end. And still leaves me without a clue...

Bye, Jojo

Am 10/15/2012 13:00, schrieb Joachim Schmitz:

When I do that it begins like this (I'm on Windows):

D:\Src\mingw-git\t>bash -x t0000-basic.sh -v -i
+ test_description='Test the very basics part #1.
...
'
+ . ./test-lib.sh
++ ORIGINAL_TERM=cygwin
++ test -z ''
+++ pwd
++ TEST_DIRECTORY=/d/Src/mingw-git/t
++ test -z ''
++ TEST_OUTPUT_DIRECTORY=/d/Src/mingw-git/t
++ GIT_BUILD_DIR=/d/Src/mingw-git/t/..
++ /d/Src/mingw-git/t/../git
++ test 1 '!=' 1
++ . /d/Src/mingw-git/t/../GIT-BUILD-OPTIONS
+++ SHELL_PATH=/bin/sh
+++ PERL_PATH=/usr/bin/perl
+++ DIFF=diff
+++ PYTHON_PATH=/usr/bin/python
+++ TAR=tar
...

It seems you need a shell that is verbose under -x.

-- Hannes

Erm, no, I left that part out...

+ . ./test-lib.sh
++ ORIGINAL_TERM=dumb
++ test -z ''
+++ pwd
++ TEST_DIRECTORY=/home/jojo/git/git/t
++ test -z ''
++ TEST_OUTPUT_DIRECTORY=/home/jojo/git/git/t
++ GIT_BUILD_DIR=/home/jojo/git/git/t/..
++ /home/jojo/git/git/t/../git
++ test 1 '!=' 1
++ . /home/jojo/git/git/t/../GIT-BUILD-OPTIONS
+++ SHELL_PATH=/bin/sh
+++ PERL_PATH=/usr/local/bin/perl
+++ DIFF=diff
+++ PYTHON_PATH=/usr/local/bin/python
+++ TAR=tar
+++ NO_CURL=
+++ USE_LIBPCRE=
+++ NO_PERL=
+++ NO_PYTHON=
+++ NO_UNIX_SOCKETS=
+++ GIT_TEST_CMP_USE_COPIED_CONTEXT=YesPlease
+++ NO_GETTEXT=
+++ GETTEXT_POISON=
++ export PERL_PATH SHELL_PATH
++ case "$GIT_TEST_TEE_STARTED, $* " in
++ LANG=C
++ LC_ALL=C
++ PAGER=cat
++ TZ=UTC
++ TERM=dumb
++ export LANG LC_ALL PAGER TERM TZ
++ EDITOR=:
+++ /usr/local/bin/perl -e '
        my @env = keys %ENV;
        my $ok = join("|", qw(
                TRACE
                DEBUG
                USE_LOOKUP
                TEST
                .*_TEST
                PROVE
                VALGRIND
                PERF_AGGREGATING_LATER
        ));
        my @vars = grep(/^GIT_/ && !/^GIT_($ok)/o, @env);
        print join("\n", @vars);
'
++ unset VISUAL EMAIL LANGUAGE COLUMNS GIT_AUTHOR_NAME GIT_MERGE_AUTOEDIT GIT_EXEC_PATH GIT_ATTR_NOSYSTEM GIT_MERGE_VERBOSITY
GIT_TEMPLATE_DIR GIT_CONFIG_NOSYSTEM GIT_AUTHOR_EMAIL GIT_COMMITTER_NAME GIT_COMMITTER_EMAIL
++ unset XDG_CONFIG_HOME
++ GIT_AUTHOR_EMAIL=author@example.com
++ GIT_AUTHOR_NAME='A U Thor'
++ GIT_COMMITTER_EMAIL=committer@example.com
++ GIT_COMMITTER_NAME='C O Mitter'
++ GIT_MERGE_VERBOSITY=5
++ GIT_MERGE_AUTOEDIT=no
++ export GIT_MERGE_VERBOSITY GIT_MERGE_AUTOEDIT
++ export GIT_AUTHOR_EMAIL GIT_AUTHOR_NAME
++ export GIT_COMMITTER_EMAIL GIT_COMMITTER_NAME
++ export EDITOR
++ expr '  ' : '.* --valgrind '
++ test -n ''
++ unset CDPATH
++ unset GREP_OPTIONS
++ case $(echo $GIT_TRACE |tr "[A-Z]" "[a-z]") in
+++ echo
+++ tr '[A-Z]' '[a-z]'
++ _x05='[0-9a-f][0-9a-f][0-9a-f][0-9a-f][0-9a-f]'
++
_x40='[0-9a-f][0-9a-f][0-9a-f][0-9a-f][0-9a-f][0-9a-f][0-9a-f][0-9a-f][0-9a-f][0-9a-f][0-9a-f][0-9a-f][0-9a-f][0-9a-f][0-9a-f][0-9a-
f][0-9a-f][0-9a-f][0-9a-f][0-9a-f][0-9a-f][0-9a-f][0-9a-f][0-9a-f][0-9a-f][0-9a-f][0-9a-f][0-9a-f][0-9a-f][0-9a-f][0-9a-f][0-9a-f][0
-9a-f][0-9a-f][0-9a-f][0-9a-f][0-9a-f][0-9a-f][0-9a-f][0-9a-f]'
++ _z40=0000000000000000000000000000000000000000
++ LF='
'
++ export _x05 _x40 _z40 LF
++ '[' xdumb '!=' xdumb ']'
++ test 2 -ne 0
++ case "$1" in
++ verbose=t
++ shift
++ test 1 -ne 0
++ case "$1" in
++ immediate=t
++ shift
++ test 0 -ne 0
++ test -n ''
++ test 'Test the very basics part #1.
The rest of the test suite does not check the basic operation of git
plumbing commands to work very carefully.  Their job is to concentrate
on tricky features that caused bugs in the past to detect regression.

This test runs very basic features, like registering things in cache, writing tree, etc.

Note that this test *deliberately* hard-codes many expected object
IDs.  When object ID computation changes, like in the previous case of
swapping compression and hashing order, the person who is making the
modification *should* take notice and update the test vectors here.
' '!=' ''
++ test '' = t
++ exec
++ exec
++ test t = t
++ exec
++ test_failure=0
++ test_count=0
++ test_fixed=0
++ test_broken=0
++ test_success=0
++ test_external_has_tap=0
++ GIT_EXIT_OK=
++ trap die EXIT
++ . /home/jojo/git/git/t/test-lib-functions.sh
+++ satisfied_prereq=' '
+++ lazily_testable_prereq=
+++ lazily_tested_prereq=
++ test -n ''
++ test -n ''
++ git_bin_dir=/home/jojo/git/git/t/../bin-wrappers
++ test -x /home/jojo/git/git/t/../bin-wrappers/git
++
PATH=/home/jojo/git/git/t/../bin-wrappers:/home/jojo/git/git/t/../bin-wrappers:/bin:/bin/unsupported:/usr/bin:/usr/ucb:/usr/local/bi
n:/usr/tandem/java/bin:/home/jojo/bin
++ GIT_EXEC_PATH=/home/jojo/git/git/t/..
++ test -n ''
++ GIT_TEMPLATE_DIR=/home/jojo/git/git/t/../templates/blt
++ unset GIT_CONFIG
++ GIT_CONFIG_NOSYSTEM=1
++ GIT_ATTR_NOSYSTEM=1
++ export PATH GIT_EXEC_PATH GIT_TEMPLATE_DIR GIT_CONFIG_NOSYSTEM GIT_ATTR_NOSYSTEM
++ test -z ''
++ test -n YesPlease
++ GIT_TEST_CMP='diff -c'
++ GITPERLLIB=/home/jojo/git/git/t/../perl/blib/lib:/home/jojo/git/git/t/../perl/blib/arch/auto/Git
++ export GITPERLLIB
++ test -d /home/jojo/git/git/t/../templates/blt
++ test -z ''
++ test -z ''
++ GITPYTHONLIB=/home/jojo/git/git/t/../git_remote_helpers/build/lib
++ export GITPYTHONLIB
++ test -d /home/jojo/git/git/t/../git_remote_helpers/build
++ test -x /home/jojo/git/git/t/../test-chmtime
+++ basename ./t0000-basic.sh .sh
++ test='trash directory.t0000-basic'
++ test -n ''
++ case "$test" in
++ TRASH_DIRECTORY='/home/jojo/git/git/t/trash directory.t0000-basic'
++ test '!' -z ''
++ remove_trash='/home/jojo/git/git/t/trash directory.t0000-basic'
++ rm -fr 'trash directory.t0000-basic'
++ HOME='/home/jojo/git/git/t/trash directory.t0000-basic'
++ export HOME
++ test -z ''
++ test_create_repo 'trash directory.t0000-basic'
++ test 1 = 1
++ repo='trash directory.t0000-basic'
++ mkdir -p 'trash directory.t0000-basic'
++ cd 'trash directory.t0000-basic'
++ /home/jojo/git/git/t/../git-init --template=/home/jojo/git/git/t/../templates/blt/
Initialized empty Git repository in /home/jojo/git/git/t/trash directory.t0000-basic/.git/
++ mv .git/hooks .git/hooks-disabled
++ cd -P 'trash directory.t0000-basic'
++ this_test=t0000-basic.sh
++ this_test=t0000
++ case $(uname -s) in
+++ uname -s
++ test_set_prereq POSIXPERM
++ satisfied_prereq=' POSIXPERM '
++ test_set_prereq BSLASHPSPEC
++ satisfied_prereq=' POSIXPERM BSLASHPSPEC '
++ test_set_prereq EXECKEEPSPID
++ satisfied_prereq=' POSIXPERM BSLASHPSPEC EXECKEEPSPID '
++ test_set_prereq NOT_MINGW
++ satisfied_prereq=' POSIXPERM BSLASHPSPEC EXECKEEPSPID NOT_MINGW '
++ COLUMNS=1
++ test 1 = 1
++ test_set_prereq COLUMNS_CAN_BE_1
++ satisfied_prereq=' POSIXPERM BSLASHPSPEC EXECKEEPSPID NOT_MINGW COLUMNS_CAN_BE_1 '
++ test -z ''
++ test_set_prereq PERL
++ satisfied_prereq=' POSIXPERM BSLASHPSPEC EXECKEEPSPID NOT_MINGW COLUMNS_CAN_BE_1 PERL '
++ test -z ''
++ test_set_prereq PYTHON
++ satisfied_prereq=' POSIXPERM BSLASHPSPEC EXECKEEPSPID NOT_MINGW COLUMNS_CAN_BE_1 PERL PYTHON '
++ test -n ''
++ test -z ''
++ test_set_prereq GETTEXT
++ satisfied_prereq=' POSIXPERM BSLASHPSPEC EXECKEEPSPID NOT_MINGW COLUMNS_CAN_BE_1 PERL PYTHON GETTEXT '
++ test -n ''
++ test_set_prereq C_LOCALE_OUTPUT
++ satisfied_prereq=' POSIXPERM BSLASHPSPEC EXECKEEPSPID NOT_MINGW COLUMNS_CAN_BE_1 PERL PYTHON GETTEXT C_LOCALE_OUTPUT '
++ test_lazy_prereq SYMLINKS '
        # test whether the filesystem supports symbolic links
        ln -s x y && test -h y
'
++ lazily_testable_prereq='SYMLINKS '
++ eval 'test_prereq_lazily_SYMLINKS=$2'
+++ test_prereq_lazily_SYMLINKS='
        # test whether the filesystem supports symbolic links
        ln -s x y && test -h y
'
++ test_lazy_prereq CASE_INSENSITIVE_FS '
        echo good >CamelCase &&
        echo bad >camelcase &&
        test "$(cat CamelCase)" != good
'
++ lazily_testable_prereq='SYMLINKS CASE_INSENSITIVE_FS '
++ eval 'test_prereq_lazily_CASE_INSENSITIVE_FS=$2'
+++ test_prereq_lazily_CASE_INSENSITIVE_FS='
        echo good >CamelCase &&
        echo bad >camelcase &&
        test "$(cat CamelCase)" != good
'
++ test_lazy_prereq UTF8_NFD_TO_NFC '
        # check whether FS converts nfd unicode to nfc
        auml=$(printf "\303\244")
        aumlcdiar=$(printf "\141\314\210")
        >"$auml" &&
        case "$(echo *)" in
        "$aumlcdiar")
                true ;;
        *)
                false ;;
        esac
'
++ lazily_testable_prereq='SYMLINKS CASE_INSENSITIVE_FS UTF8_NFD_TO_NFC '
++ eval 'test_prereq_lazily_UTF8_NFD_TO_NFC=$2'
+++ test_prereq_lazily_UTF8_NFD_TO_NFC='
        # check whether FS converts nfd unicode to nfc
        auml=$(printf "\303\244")
        aumlcdiar=$(printf "\141\314\210")
        >"$auml" &&
        case "$(echo *)" in
        "$aumlcdiar")
                true ;;
        *)
                false ;;
        esac
'
++ test -w /
++ test_set_prereq SANITY
++ satisfied_prereq=' POSIXPERM BSLASHPSPEC EXECKEEPSPID NOT_MINGW COLUMNS_CAN_BE_1 PERL PYTHON GETTEXT C_LOCALE_OUTPUT SANITY '
+ test_expect_success '.git/objects should be empty after git init in an empty repo' '
        find .git/objects -type f -print >should-be-empty &&
        test_line_count = 0 should-be-empty
'
+ test 2 = 3
+ test_prereq=
+ test 2 = 2
+ export test_prereq
+ test_skip '.git/objects should be empty after git init in an empty repo' '
        find .git/objects -type f -print >should-be-empty &&
        test_line_count = 0 should-be-empty
'
+ test_count=1
+ to_skip=
+ test -z ''
+ test -n ''
+ case "$to_skip" in
+ false
+ say 'expecting success:
        find .git/objects -type f -print >should-be-empty &&
        test_line_count = 0 should-be-empty
'
+ say_color info 'expecting success:
        find .git/objects -type f -print >should-be-empty &&
        test_line_count = 0 should-be-empty
'
+ test -z info
+ shift
+ echo 'expecting success:
        find .git/objects -type f -print >should-be-empty &&
        test_line_count = 0 should-be-empty
'
expecting success:
        find .git/objects -type f -print >should-be-empty &&
        test_line_count = 0 should-be-empty

+ test_run_ '
        find .git/objects -type f -print >should-be-empty &&
        test_line_count = 0 should-be-empty
'
+ test_cleanup=:
+ expecting_failure=
+ test_eval_ '
        find .git/objects -type f -print >should-be-empty &&
        test_line_count = 0 should-be-empty
'
+ eval '
        find .git/objects -type f -print >should-be-empty &&
        test_line_count = 0 should-be-empty
'
++ find .git/objects -type f -print
++ test_line_count = 0 should-be-empty
++ test 3 '!=' 3
+++ wc -l
++ test 0 = 0
+ eval_ret=0
+ test -z t
+ test 0 = 0
+ setup_malloc_check
+ MALLOC_CHECK_=3
+ MALLOC_PERTURB_=165
+ export MALLOC_CHECK_ MALLOC_PERTURB_
+ test_eval_ :
+ eval :
++
+ teardown_malloc_check
+ unset MALLOC_CHECK_ MALLOC_PERTURB_
+ test t = t
+ test -n ''
+ return 0
+ test_ok_ '.git/objects should be empty after git init in an empty repo'
+ test_success=1
+ say_color '' 'ok 1 - .git/objects should be empty after git init in an empty repo'
+ test -z ''
+ test -n ''
+ shift
+ echo 'ok 1 - .git/objects should be empty after git init in an empty repo'
ok 1 - .git/objects should be empty after git init in an empty repo
+ echo ''

+ test_expect_success '.git/objects should have 3 subdirectories' '
        find .git/objects -type d -print >full-of-directories &&
        test_line_count = 3 full-of-directories
'
+ test 2 = 3
+ test_prereq=
...

Bye, Jojo

Am 10/15/2012 13:37, schrieb Joachim Schmitz:

This is the key line. If it is 'eval_ret=1' (or other non-zero value), then the test failed, and the lines above it usually indicate where in the test snippet the failure occurred.

-- Hannes

...
++ mkdir failing-cleanup
++ cd failing-cleanup
++ cat
++ chmod +x failing-cleanup.sh
++ test_must_fail ./failing-cleanup.sh
+ eval_ret=1
+ test -z t
+ test 1 = 0
+ test -n ''
+ test t = t
+ test -n ''
+ return 1
+ test_failure_ 'tests clean up even on failures' '
...

This part?

Am 10/15/2012 13:58, schrieb Joachim Schmitz:

I wonder why the log does not show the commands of function test_must_fail. Is there a 'set +x' hidden somewhere? Run ./failing-cleanup.sh manually. Check its exit code (it should be non-zero, but not something strange like 127 and above; see test_must_fail()) and dig further from there. I'll stop here.

-- Hannes

Johannes Sixt <j.sixt@viscovery.net> writes:

That's because stderr is redirected.

Andreas.

cat err ++ ./failing-cleanup.sh ++ exit_code=0 ++ test 0 = 0 ++ echo 'test_must_fail: command succeeded: ./failing-cleanup.sh' test_must_fail: command succeeded: ./failing-cleanup.sh ++ return 1

That test (as well as quite a few more) do pass when using bash rather than our sh (which really is a ksh)

Bye, Jojo

It returns 0.