# [PATCH v1 2/3] travis-ci: disable verbose test output 10 messages from 2016-05-22 to 2016-05-26. Participants: larsxschneider@gmail.com, Christian Couder, Lars Schneider, Pranit Bauva, Junio C Hamano, Jeff King. Thread: https://gitlist.dev/t/42411 ## larsxschneider@gmail.com, 2016-05-22 11:00 Subject: [PATCH v1 0/3] travis-ci: run Git bisect on failed tests Message-ID: <1463914856-64745-1-git-send-email-larsxschneider@gmail.com> URL: https://gitlist.dev/e/1463914856-64745-1-git-send-email-larsxschneider%40gmail.com ``` From: Lars Schneider This patch series run Git bisect on failing tests on Travis CI. Example output on Travis CI: https://travis-ci.org/larsxschneider/git/jobs/132049871 Plaintext example output: https://s3.amazonaws.com/archive.travis-ci.org/jobs/132049871/log.txt Please scroll all the way down to see the bisect output. Cheers, Lars Lars Schneider (3): travis-ci: move "after_failure" code to dedicated file in /ci travis-ci: disable verbose test output travis-ci: run Git bisect on failed tests .travis.yml | 14 ++-------- ci/test-report.sh | 84 +++++++++++++++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 86 insertions(+), 12 deletions(-) create mode 100755 ci/test-report.sh -- 2.5.1 ``` ## larsxschneider@gmail.com, 2016-05-22 11:00 Subject: [PATCH v1 1/3] travis-ci: move "after_failure" code to dedicated file in /ci Message-ID: <1463914856-64745-2-git-send-email-larsxschneider@gmail.com> URL: https://gitlist.dev/e/1463914856-64745-2-git-send-email-larsxschneider%40gmail.com In-Reply-To: <1463914856-64745-1-git-send-email-larsxschneider@gmail.com> ``` From: Lars Schneider Move the code and adjust it to the Git shell script coding guidelines. Signed-off-by: Lars Schneider --- .travis.yml | 12 +----------- ci/test-report.sh | 19 +++++++++++++++++++ 2 files changed, 20 insertions(+), 11 deletions(-) create mode 100755 ci/test-report.sh diff --git a/.travis.yml b/.travis.yml index adab5b8..a93ecb3 100644 --- a/.travis.yml +++ b/.travis.yml @@ -101,17 +101,7 @@ script: make --quiet test after_failure: - > : '<-- Click here to see detailed test output! '; - for TEST_EXIT in t/test-results/*.exit; - do - if [ "$(cat "$TEST_EXIT")" != "0" ]; - then - TEST_OUT="${TEST_EXIT%exit}out"; - echo "------------------------------------------------------------------------"; - echo "$(tput setaf 1)${TEST_OUT}...$(tput sgr0)"; - echo "------------------------------------------------------------------------"; - cat "${TEST_OUT}"; - fi; - done; + ./ci/test-report.sh notifications: email: false diff --git a/ci/test-report.sh b/ci/test-report.sh new file mode 100755 index 0000000..d08a999 --- /dev/null +++ b/ci/test-report.sh @@ -0,0 +1,19 @@ +#!/bin/sh +# +# Print test results +# +for TEST_EXIT in t/test-results/*.exit +do + if test "$(cat "$TEST_EXIT")" != "0" + then + TEST="${TEST_EXIT%.exit}" + TEST_OUT="${TEST}.out" + echo "------------------------------------------------------------------------" + echo " $(tput setaf 1)${TEST} Output$(tput sgr0)" + echo "------------------------------------------------------------------------" + cat "$TEST_OUT" + echo "" + echo "" + fi +done + -- 2.5.1 ``` ## larsxschneider@gmail.com, 2016-05-22 11:00 Subject: [PATCH v1 2/3] travis-ci: disable verbose test output Message-ID: <1463914856-64745-3-git-send-email-larsxschneider@gmail.com> URL: https://gitlist.dev/e/1463914856-64745-3-git-send-email-larsxschneider%40gmail.com In-Reply-To: <1463914856-64745-1-git-send-email-larsxschneider@gmail.com> ``` From: Lars Schneider The verbose output clutters the Travis CI webview and is not really useful since test debugging usually happens on a local machine. Signed-off-by: Lars Schneider --- .travis.yml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/.travis.yml b/.travis.yml index a93ecb3..81d2027 100644 --- a/.travis.yml +++ b/.travis.yml @@ -29,7 +29,7 @@ env: - LINUX_GIT_LFS_VERSION="1.2.0" - DEFAULT_TEST_TARGET=prove - GIT_PROVE_OPTS="--timer --jobs 3 --state=failed,slow,save" - - GIT_TEST_OPTS="--verbose --tee" + - GIT_TEST_OPTS="--tee" - GIT_TEST_CLONE_2GB=YesPlease # t9810 occasionally fails on Travis CI OS X # t9816 occasionally fails with "TAP out of sequence errors" on Travis CI OS X -- 2.5.1 ``` ## larsxschneider@gmail.com, 2016-05-22 11:00 Subject: [PATCH v1 3/3] travis-ci: run Git bisect on failed tests Message-ID: <1463914856-64745-4-git-send-email-larsxschneider@gmail.com> URL: https://gitlist.dev/e/1463914856-64745-4-git-send-email-larsxschneider%40gmail.com In-Reply-To: <1463914856-64745-1-git-send-email-larsxschneider@gmail.com> ``` From: Lars Schneider Junio usually pushes many commits at once to the public "pu"/"next"/ "master" branches. If a test fails then it is not obvious what commit caused the failure. Therefore we run Git bisect with the merge base between the failing rev and its more stable branch ("next" for "pu", "master" for "next", and "maint" for "master") as good ref to find the offending commit. This is only enabled on "github.com/git/git" because there we can assume that all relevant branches are up to date. Signed-off-by: Lars Schneider --- .travis.yml | 2 +- ci/test-report.sh | 65 +++++++++++++++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 66 insertions(+), 1 deletion(-) diff --git a/.travis.yml b/.travis.yml index 81d2027..922807b 100644 --- a/.travis.yml +++ b/.travis.yml @@ -101,7 +101,7 @@ script: make --quiet test after_failure: - > : '<-- Click here to see detailed test output! '; - ./ci/test-report.sh + ./ci/test-report.sh $TRAVIS_REPO_SLUG $TRAVIS_BRANCH; notifications: email: false diff --git a/ci/test-report.sh b/ci/test-report.sh index d08a999..8f7adad 100755 --- a/ci/test-report.sh +++ b/ci/test-report.sh @@ -1,5 +1,12 @@ #!/bin/sh # +# Print test results and run Git bisect on failed tests. +# +REPO_ORG_NAME=$1 +CURRENT_BRANCH_NAME=$2 + + +# # Print test results # for TEST_EXIT in t/test-results/*.exit @@ -17,3 +24,61 @@ do fi done + +# +# Run Git bisect +# +run_bisect () { + TEST_SCRIPT=$1 + BAD_REV=$2 + GOOD_RV=$3 + TMPDIR=$(mktemp -d -t "ci-report-bisect-XXXXXX" 2>/dev/null) + cat > "$TMPDIR/bisect-run.sh" </dev/null 2>&1 + then + cd t && ./$TEST_SCRIPT.sh >/dev/null 2>&1 + else + # If the test file does not exist or the build fails then tell + # Git bisect to skip the commit. + exit 125 + fi + git bisect reset >/dev/null 2>&1 +} + +case "$CURRENT_BRANCH_NAME" in + master) STABLE_BRANCH="maint";; + next) STABLE_BRANCH="master";; + pu) STABLE_BRANCH="next";; +esac + +if test "$REPO_ORG_NAME" = "git/git" && test -n $STABLE_BRANCH +then + BAD_REV=$(git rev-parse HEAD) + + # Travis CI clones are shallow. It is possible that the last good revision + # was not fetched, yet. Therefore we need to fetch all commits on the + # stable branch. + git config remote.origin.fetch "+refs/heads/$STABLE_BRANCH:refs/remotes/origin/$STABLE_BRANCH" + git fetch --unshallow --quiet + LAST_GOOD_REV=$(git merge-base $BAD_REV "remotes/origin/$STABLE_BRANCH") + + for TEST_EXIT in t/test-results/*.exit + do + if test "$(cat "$TEST_EXIT")" != "0" + then + TEST="${TEST_EXIT%.exit}" + TEST_SCRIPT=${TEST#t/test-results/} + echo "------------------------------------------------------------------------" + echo " $(tput setaf 1)${TEST} Bisect$(tput sgr0)" + echo "------------------------------------------------------------------------" + run_bisect $TEST_SCRIPT $BAD_REV $LAST_GOOD_REV + echo "" + echo "" + fi + done +fi -- 2.5.1 ``` ## Christian Couder, 2016-05-22 15:35 Subject: Re: [PATCH v1 3/3] travis-ci: run Git bisect on failed tests Message-ID: URL: https://gitlist.dev/e/CAP8UFD3TJ7mJEu2xCXhrspHY1v02uTonUGdszZbcQGf9ne9_Cg%40mail.gmail.com In-Reply-To: <1463914856-64745-4-git-send-email-larsxschneider@gmail.com> ``` On Sun, May 22, 2016 at 1:00 PM, wrote: [...] > +# > +# Run Git bisect > +# > +run_bisect () { > + TEST_SCRIPT=$1 > + BAD_REV=$2 > + GOOD_RV=$3 > + TMPDIR=$(mktemp -d -t "ci-report-bisect-XXXXXX" 2>/dev/null) > + cat > "$TMPDIR/bisect-run.sh" < + > +EOF > + chmod +x "$TMPDIR/bisect-run.sh" > + git bisect start $BAD_REV $GOOD_RV > + git bisect run "$TMPDIR/bisect-run.sh" > + if test -e ./t/$TEST_SCRIPT.sh && make --jobs=2 >/dev/null 2>&1 > + then > + cd t && ./$TEST_SCRIPT.sh >/dev/null 2>&1 > + else > + # If the test file does not exist or the build fails then tell > + # Git bisect to skip the commit. > + exit 125 > + fi Shouldn't all the above "if ... fi" be in the here document creating "$TMPDIR/bisect-run.sh"? > + git bisect reset >/dev/null 2>&1 > +} ``` ## Lars Schneider, 2016-05-22 16:35 Subject: Re: [PATCH v1 3/3] travis-ci: run Git bisect on failed tests Message-ID: URL: https://gitlist.dev/e/DEFC086F-6206-4598-8355-0394C6070F59%40gmail.com In-Reply-To: ``` > On 22 May 2016, at 17:35, Christian Couder wrote: > > On Sun, May 22, 2016 at 1:00 PM, wrote: > > [...] > >> +# >> +# Run Git bisect >> +# >> +run_bisect () { >> + TEST_SCRIPT=$1 >> + BAD_REV=$2 >> + GOOD_RV=$3 >> + TMPDIR=$(mktemp -d -t "ci-report-bisect-XXXXXX" 2>/dev/null) >> + cat > "$TMPDIR/bisect-run.sh" <> + >> +EOF >> + chmod +x "$TMPDIR/bisect-run.sh" >> + git bisect start $BAD_REV $GOOD_RV >> + git bisect run "$TMPDIR/bisect-run.sh" >> + if test -e ./t/$TEST_SCRIPT.sh && make --jobs=2 >/dev/null 2>&1 >> + then >> + cd t && ./$TEST_SCRIPT.sh >/dev/null 2>&1 >> + else >> + # If the test file does not exist or the build fails then tell >> + # Git bisect to skip the commit. >> + exit 125 >> + fi > > Shouldn't all the above "if ... fi" be in the here document creating > "$TMPDIR/bisect-run.sh"? Ohh. Absolutely... I wonder what happened. I'll post a v2. Thank you, Lars > >> + git bisect reset >/dev/null 2>&1 >> +} ``` ## Pranit Bauva, 2016-05-22 17:21 Subject: Re: [PATCH v1 3/3] travis-ci: run Git bisect on failed tests Message-ID: URL: https://gitlist.dev/e/CAFZEwPOuCRJFx0wH5YHKxWNmjkvi2R6s5G0Bw%2BWrnHywT3cR0g%40mail.gmail.com In-Reply-To: <1463914856-64745-4-git-send-email-larsxschneider@gmail.com> ``` Hey Lars, On Sun, May 22, 2016 at 4:30 PM, wrote: > From: Lars Schneider > > Junio usually pushes many commits at once to the public "pu"/"next"/ > "master" branches. If a test fails then it is not obvious what commit > caused the failure. Therefore we run Git bisect with the merge base > between the failing rev and its more stable branch ("next" for "pu", > "master" for "next", and "maint" for "master") as good ref to find the > offending commit. This is only enabled on "github.com/git/git" because > there we can assume that all relevant branches are up to date. > > Signed-off-by: Lars Schneider > --- > .travis.yml | 2 +- > ci/test-report.sh | 65 +++++++++++++++++++++++++++++++++++++++++++++++++++++++ > 2 files changed, 66 insertions(+), 1 deletion(-) > > diff --git a/.travis.yml b/.travis.yml > index 81d2027..922807b 100644 > --- a/.travis.yml > +++ b/.travis.yml > @@ -101,7 +101,7 @@ script: make --quiet test > after_failure: > - > > : '<-- Click here to see detailed test output! '; > - ./ci/test-report.sh > + ./ci/test-report.sh $TRAVIS_REPO_SLUG $TRAVIS_BRANCH; > > notifications: > email: false > diff --git a/ci/test-report.sh b/ci/test-report.sh > index d08a999..8f7adad 100755 > --- a/ci/test-report.sh > +++ b/ci/test-report.sh > @@ -1,5 +1,12 @@ > #!/bin/sh > # > +# Print test results and run Git bisect on failed tests. > +# > +REPO_ORG_NAME=$1 > +CURRENT_BRANCH_NAME=$2 > + > + > +# > # Print test results > # > for TEST_EXIT in t/test-results/*.exit > @@ -17,3 +24,61 @@ do > fi > done > > + > +# > +# Run Git bisect > +# > +run_bisect () { > + TEST_SCRIPT=$1 > + BAD_REV=$2 > + GOOD_RV=$3 > + TMPDIR=$(mktemp -d -t "ci-report-bisect-XXXXXX" 2>/dev/null) > + cat > "$TMPDIR/bisect-run.sh" <"$TMPDIR..." ' > + > +EOF > + chmod +x "$TMPDIR/bisect-run.sh" > + git bisect start $BAD_REV $GOOD_RV > + git bisect run "$TMPDIR/bisect-run.sh" > + if test -e ./t/$TEST_SCRIPT.sh && make --jobs=2 >/dev/null 2>&1 > + then > + cd t && ./$TEST_SCRIPT.sh >/dev/null 2>&1 > + else > + # If the test file does not exist or the build fails then tell > + # Git bisect to skip the commit. > + exit 125 > + fi > + git bisect reset >/dev/null 2>&1 > +} > + > +case "$CURRENT_BRANCH_NAME" in > + master) STABLE_BRANCH="maint";; > + next) STABLE_BRANCH="master";; > + pu) STABLE_BRANCH="next";; > +esac > + > +if test "$REPO_ORG_NAME" = "git/git" && test -n $STABLE_BRANCH > +then > + BAD_REV=$(git rev-parse HEAD) > + > + # Travis CI clones are shallow. It is possible that the last good revision > + # was not fetched, yet. Therefore we need to fetch all commits on the > + # stable branch. > + git config remote.origin.fetch "+refs/heads/$STABLE_BRANCH:refs/remotes/origin/$STABLE_BRANCH" > + git fetch --unshallow --quiet > + LAST_GOOD_REV=$(git merge-base $BAD_REV "remotes/origin/$STABLE_BRANCH") > + > + for TEST_EXIT in t/test-results/*.exit > + do > + if test "$(cat "$TEST_EXIT")" != "0" > + then > + TEST="${TEST_EXIT%.exit}" > + TEST_SCRIPT=${TEST#t/test-results/} > + echo "------------------------------------------------------------------------" > + echo " $(tput setaf 1)${TEST} Bisect$(tput sgr0)" > + echo "------------------------------------------------------------------------" > + run_bisect $TEST_SCRIPT $BAD_REV $LAST_GOOD_REV > + echo "" > + echo "" > + fi > + done > +fi Regards, Pranit Bauva ``` ## Junio C Hamano, 2016-05-23 01:22 Subject: Re: [PATCH v1 3/3] travis-ci: run Git bisect on failed tests Message-ID: URL: https://gitlist.dev/e/xmqq1t4tbmbp.fsf%40gitster.mtv.corp.google.com In-Reply-To: <1463914856-64745-4-git-send-email-larsxschneider@gmail.com> ``` larsxschneider@gmail.com writes: > Junio usually pushes many commits at once to the public "pu"/"next"/ > "master" branches. If a test fails then it is not obvious what commit > caused the failure. Therefore we run Git bisect with the merge base > between the failing rev and its more stable branch ("next" for "pu", > "master" for "next", and "maint" for "master") as good ref to find the > offending commit. This is only enabled on "github.com/git/git" because > there we can assume that all relevant branches are up to date. Just FYI, 'git log next..pu' is not a very interesting range unless you also use the "--no-merges" option. Is there a good way to tell the CI that, instead of testing 'pu' (or a specific branch in general), test all new commits that appear on the first-parent chain between 'master'..'pu'? It would be ideal if CI can do this every time 'master' and 'pu' gets updated. - Find all commits that appear on the first-parent chain between 'master..pu'. - For each of them: - Test it. This tests the merge result and can catch breakages introduced by mismerging. - Also test its second parent if it is a merge. This tests the tip of each topic branch in isolation. CI gets a bonus point if it can remember that the tip of the topic did not move and the commit already passed the test to skip it ;-) - Optionally, for those topics that failed the "tip of the topic branch" test, "bisect master..$topic" to see where it breaks. The thing is, "bisect" works only when you have a _single_ cause of breakage, but because 'pu' is a pile of unsorted raw material yet to be sifted into dirt and gem, once you see a failure at the tip of 'pu', mechanical 'bisect run' may not be a very useful tool, as you do not know if there is only one breakage you are looking for. Because the unit of integration we use is a topic branch (not an individual commit on a topic branch), "this topic is broken", and "this topic itself may be good, but merging it breaks 'pu'" are far more interesting and relevant piece of information than "this commit is what broke 'pu'". ``` ## Lars Schneider, 2016-05-23 08:36 Subject: Re: [PATCH v1 3/3] travis-ci: run Git bisect on failed tests Message-ID: URL: https://gitlist.dev/e/BCC937A6-27E4-4849-80C7-82EC155F08C1%40gmail.com In-Reply-To: ``` > On 22 May 2016, at 19:21, Pranit Bauva wrote: > > Hey Lars, > > On Sun, May 22, 2016 at 4:30 PM, wrote: >> From: Lars Schneider >> ... >> + >> +# >> +# Run Git bisect >> +# >> +run_bisect () { >> + TEST_SCRIPT=$1 >> + BAD_REV=$2 >> + GOOD_RV=$3 >> + TMPDIR=$(mktemp -d -t "ci-report-bisect-XXXXXX" 2>/dev/null) >> + cat > "$TMPDIR/bisect-run.sh" < > If you are doing a re-roll, then you could probably fix the style issue. > ' cat >"$TMPDIR..." ' Good eye :-) I will fix this, too. Thank you, Lars ``` ## Jeff King, 2016-05-26 04:54 Subject: Re: [PATCH v1 2/3] travis-ci: disable verbose test output Message-ID: <20160526045455.GD6756@sigill.intra.peff.net> URL: https://gitlist.dev/e/20160526045455.GD6756%40sigill.intra.peff.net In-Reply-To: <1463914856-64745-3-git-send-email-larsxschneider@gmail.com> ``` On Sun, May 22, 2016 at 01:00:55PM +0200, larsxschneider@gmail.com wrote: > From: Lars Schneider > > The verbose output clutters the Travis CI webview and is not really > useful since test debugging usually happens on a local machine. I have not really been using the Travis CI results, so perhaps my opinion does not count. But in other systems, I have found that the more verbose the CI output, the better, simply because you will inevitably be faced with a test that breaks on CI and not your local machine, and you will have no way to get more details. I don't know if Travis provides a better way to hide the output in the non-failing cases. -Peff ```