threads / patch / 42411

patch, 3 partstravis-ci: disable verbose test output

Subject: [PATCH v1 2/3] travis-ci: disable verbose test output

## tl;dr

10 messages between May 22, 2016 and May 26, 2016. Diffs are folded; open one to read it.

replies: 9people: 5as markdown or json

larsxschneider@gmail.com· May 22, 2016, 11:00 UTC · lore

[PATCH v1 0/3] travis-ci: run Git bisect on failed tests

From: Lars Schneider <larsxschneider@gmail.com>
This patch series run Git bisect on failing tests on Travis CI.

Example output on Travis CI: https://travis-ci.org/larsxschneider/git/jobs/132049871

Plaintext example output: https://s3.amazonaws.com/archive.travis-ci.org/jobs/132049871/log.txt

Please scroll all the way down to see the bisect output.

Cheers, Lars

Lars Schneider (3):
  travis-ci: move "after_failure" code to dedicated file in /ci
  travis-ci: disable verbose test output
  travis-ci: run Git bisect on failed tests
 .travis.yml       | 14 ++--------
 ci/test-report.sh | 84 +++++++++++++++++++++++++++++++++++++++++++++++++++++++
 2 files changed, 86 insertions(+), 12 deletions(-)
 create mode 100755 ci/test-report.sh

-- 2.5.1

larsxschneider@gmail.com· May 22, 2016, 11:00 UTC · re: larsxschneider@gmail.com · lore

[PATCH v1 1/3] travis-ci: move "after_failure" code to dedicated file in /ci

From: Lars Schneider <larsxschneider@gmail.com>
Move the code and adjust it to the Git shell script coding guidelines.
Signed-off-by: Lars Schneider <larsxschneider@gmail.com>
---
 .travis.yml       | 12 +-----------
 ci/test-report.sh | 19 +++++++++++++++++++
 2 files changed, 20 insertions(+), 11 deletions(-)
 create mode 100755 ci/test-report.sh
Show changes to 2 files +20 −11

.travis.yml, ci/test-report.sh

diff --git a/.travis.yml b/.travis.yml
index adab5b8..a93ecb3 100644
--- a/.travis.yml
+++ b/.travis.yml
@@ -101,17 +101,7 @@ script: make --quiet test
 after_failure:
   - >
     : '<-- Click here to see detailed test output!                                                        ';
-    for TEST_EXIT in t/test-results/*.exit;
-    do
-      if [ "$(cat "$TEST_EXIT")" != "0" ];
-      then
-        TEST_OUT="${TEST_EXIT%exit}out";
-        echo "------------------------------------------------------------------------";
-        echo "$(tput setaf 1)${TEST_OUT}...$(tput sgr0)";
-        echo "------------------------------------------------------------------------";
-        cat "${TEST_OUT}";
-      fi;
-    done;
+    ./ci/test-report.sh
 
 notifications:
   email: false
diff --git a/ci/test-report.sh b/ci/test-report.sh
new file mode 100755
index 0000000..d08a999
--- /dev/null
+++ b/ci/test-report.sh
@@ -0,0 +1,19 @@
+#!/bin/sh
+#
+# Print test results
+#
+for TEST_EXIT in t/test-results/*.exit
+do
+	if test "$(cat "$TEST_EXIT")" != "0"
+	then
+		TEST="${TEST_EXIT%.exit}"
+		TEST_OUT="${TEST}.out"
+		echo "------------------------------------------------------------------------"
+		echo "  $(tput setaf 1)${TEST} Output$(tput sgr0)"
+		echo "------------------------------------------------------------------------"
+		cat "$TEST_OUT"
+		echo ""
+		echo ""
+	fi
+done
+

-- 
2.5.1
larsxschneider@gmail.com· May 22, 2016, 11:00 UTC · re: larsxschneider@gmail.com · lore
From: Lars Schneider <larsxschneider@gmail.com>

The verbose output clutters the Travis CI webview and is not really useful since test debugging usually happens on a local machine.

Signed-off-by: Lars Schneider <larsxschneider@gmail.com>
---
 .travis.yml | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
Show changes to .travis.yml +1 −1
diff --git a/.travis.yml b/.travis.yml
index a93ecb3..81d2027 100644
--- a/.travis.yml
+++ b/.travis.yml
@@ -29,7 +29,7 @@ env:
     - LINUX_GIT_LFS_VERSION="1.2.0"
     - DEFAULT_TEST_TARGET=prove
     - GIT_PROVE_OPTS="--timer --jobs 3 --state=failed,slow,save"
-    - GIT_TEST_OPTS="--verbose --tee"
+    - GIT_TEST_OPTS="--tee"
     - GIT_TEST_CLONE_2GB=YesPlease
     # t9810 occasionally fails on Travis CI OS X
     # t9816 occasionally fails with "TAP out of sequence errors" on Travis CI OS X

-- 
2.5.1
Jeff King· May 26, 2016, 04:54 UTC · re: larsxschneider@gmail.com · lore

Re: [PATCH v1 2/3] travis-ci: disable verbose test output

On Sun, May 22, 2016 at 01:00:55PM +0200, larsxschneider@gmail.com wrote:
> From: Lars Schneider <larsxschneider@gmail.com>
> 
> The verbose output clutters the Travis CI webview and is not really
> useful since test debugging usually happens on a local machine.

I have not really been using the Travis CI results, so perhaps my opinion does not count. But in other systems, I have found that the more verbose the CI output, the better, simply because you will inevitably be faced with a test that breaks on CI and not your local machine, and you will have no way to get more details.

I don't know if Travis provides a better way to hide the output in the non-failing cases.

-Peff
larsxschneider@gmail.com· May 22, 2016, 11:00 UTC · re: larsxschneider@gmail.com · lore

[PATCH v1 3/3] travis-ci: run Git bisect on failed tests

From: Lars Schneider <larsxschneider@gmail.com>

Junio usually pushes many commits at once to the public "pu"/"next"/ "master" branches. If a test fails then it is not obvious what commit caused the failure. Therefore we run Git bisect with the merge base between the failing rev and its more stable branch ("next" for "pu", "master" for "next", and "maint" for "master") as good ref to find the offending commit. This is only enabled on "github.com/git/git" because there we can assume that all relevant branches are up to date.

Signed-off-by: Lars Schneider <larsxschneider@gmail.com>
---
 .travis.yml       |  2 +-
 ci/test-report.sh | 65 +++++++++++++++++++++++++++++++++++++++++++++++++++++++
 2 files changed, 66 insertions(+), 1 deletion(-)
Show changes to 2 files +66 −1

.travis.yml, ci/test-report.sh

diff --git a/.travis.yml b/.travis.yml
index 81d2027..922807b 100644
--- a/.travis.yml
+++ b/.travis.yml
@@ -101,7 +101,7 @@ script: make --quiet test
 after_failure:
   - >
     : '<-- Click here to see detailed test output!                                                        ';
-    ./ci/test-report.sh
+    ./ci/test-report.sh $TRAVIS_REPO_SLUG $TRAVIS_BRANCH;
 
 notifications:
   email: false
diff --git a/ci/test-report.sh b/ci/test-report.sh
index d08a999..8f7adad 100755
--- a/ci/test-report.sh
+++ b/ci/test-report.sh
@@ -1,5 +1,12 @@
 #!/bin/sh
 #
+# Print test results and run Git bisect on failed tests.
+#
+REPO_ORG_NAME=$1
+CURRENT_BRANCH_NAME=$2
+
+
+#
 # Print test results
 #
 for TEST_EXIT in t/test-results/*.exit
@@ -17,3 +24,61 @@ do
 	fi
 done
 
+
+#
+# Run Git bisect
+#
+run_bisect () {
+	TEST_SCRIPT=$1
+	BAD_REV=$2
+	GOOD_RV=$3
+	TMPDIR=$(mktemp -d -t "ci-report-bisect-XXXXXX" 2>/dev/null)
+	cat > "$TMPDIR/bisect-run.sh" <<EOF
+
+EOF
+	chmod +x "$TMPDIR/bisect-run.sh"
+	git bisect start $BAD_REV $GOOD_RV
+	git bisect run "$TMPDIR/bisect-run.sh"
+	if test -e ./t/$TEST_SCRIPT.sh && make --jobs=2 >/dev/null 2>&1
+	then
+		cd t && ./$TEST_SCRIPT.sh >/dev/null 2>&1
+	else
+		# If the test file does not exist or the build fails then tell
+		# Git bisect to skip the commit.
+		exit 125
+	fi
+	git bisect reset >/dev/null 2>&1
+}
+
+case "$CURRENT_BRANCH_NAME" in
+	master) STABLE_BRANCH="maint";;
+	next)   STABLE_BRANCH="master";;
+	pu)     STABLE_BRANCH="next";;
+esac
+
+if test "$REPO_ORG_NAME" = "git/git" && test -n $STABLE_BRANCH
+then
+	BAD_REV=$(git rev-parse HEAD)
+
+	# Travis CI clones are shallow. It is possible that the last good revision
+	# was not fetched, yet. Therefore we need to fetch all commits on the
+	# stable branch.
+	git config remote.origin.fetch "+refs/heads/$STABLE_BRANCH:refs/remotes/origin/$STABLE_BRANCH"
+	git fetch --unshallow --quiet
+	LAST_GOOD_REV=$(git merge-base $BAD_REV "remotes/origin/$STABLE_BRANCH")
+
+	for TEST_EXIT in t/test-results/*.exit
+	do
+		if test "$(cat "$TEST_EXIT")" != "0"
+		then
+			TEST="${TEST_EXIT%.exit}"
+			TEST_SCRIPT=${TEST#t/test-results/}
+			echo "------------------------------------------------------------------------"
+			echo "  $(tput setaf 1)${TEST} Bisect$(tput sgr0)"
+			echo "------------------------------------------------------------------------"
+			run_bisect $TEST_SCRIPT $BAD_REV $LAST_GOOD_REV
+			echo ""
+			echo ""
+		fi
+	done
+fi

-- 
2.5.1
Christian Couder· May 22, 2016, 15:35 UTC · re: larsxschneider@gmail.com · lore

Re: [PATCH v1 3/3] travis-ci: run Git bisect on failed tests

On Sun, May 22, 2016 at 1:00 PM,  <larsxschneider@gmail.com> wrote:
[...]
Show 22 quoted lines
> +#
> +# Run Git bisect
> +#
> +run_bisect () {
> +       TEST_SCRIPT=$1
> +       BAD_REV=$2
> +       GOOD_RV=$3
> +       TMPDIR=$(mktemp -d -t "ci-report-bisect-XXXXXX" 2>/dev/null)
> +       cat > "$TMPDIR/bisect-run.sh" <<EOF
> +
> +EOF
> +       chmod +x "$TMPDIR/bisect-run.sh"
> +       git bisect start $BAD_REV $GOOD_RV
> +       git bisect run "$TMPDIR/bisect-run.sh"
> +       if test -e ./t/$TEST_SCRIPT.sh && make --jobs=2 >/dev/null 2>&1
> +       then
> +               cd t && ./$TEST_SCRIPT.sh >/dev/null 2>&1
> +       else
> +               # If the test file does not exist or the build fails then tell
> +               # Git bisect to skip the commit.
> +               exit 125
> +       fi

Shouldn't all the above "if ... fi" be in the here document creating "$TMPDIR/bisect-run.sh"?

> +       git bisect reset >/dev/null 2>&1
> +}
Lars Schneider· May 22, 2016, 16:35 UTC · re: Christian Couder · lore

Re: [PATCH v1 3/3] travis-ci: run Git bisect on failed tests

Show 31 quoted lines
> On 22 May 2016, at 17:35, Christian Couder <christian.couder@gmail.com> wrote:
> 
> On Sun, May 22, 2016 at 1:00 PM,  <larsxschneider@gmail.com> wrote:
> 
> [...]
> 
>> +#
>> +# Run Git bisect
>> +#
>> +run_bisect () {
>> +       TEST_SCRIPT=$1
>> +       BAD_REV=$2
>> +       GOOD_RV=$3
>> +       TMPDIR=$(mktemp -d -t "ci-report-bisect-XXXXXX" 2>/dev/null)
>> +       cat > "$TMPDIR/bisect-run.sh" <<EOF
>> +
>> +EOF
>> +       chmod +x "$TMPDIR/bisect-run.sh"
>> +       git bisect start $BAD_REV $GOOD_RV
>> +       git bisect run "$TMPDIR/bisect-run.sh"
>> +       if test -e ./t/$TEST_SCRIPT.sh && make --jobs=2 >/dev/null 2>&1
>> +       then
>> +               cd t && ./$TEST_SCRIPT.sh >/dev/null 2>&1
>> +       else
>> +               # If the test file does not exist or the build fails then tell
>> +               # Git bisect to skip the commit.
>> +               exit 125
>> +       fi
> 
> Shouldn't all the above "if ... fi" be in the here document creating
> "$TMPDIR/bisect-run.sh"?
Ohh. Absolutely... I wonder what happened. I'll post a v2.

Thank you, Lars

> 
>> +       git bisect reset >/dev/null 2>&1
>> +}
Pranit Bauva· May 22, 2016, 17:21 UTC · re: larsxschneider@gmail.com · lore

Re: [PATCH v1 3/3] travis-ci: run Git bisect on failed tests

Hey Lars,
On Sun, May 22, 2016 at 4:30 PM,  <larsxschneider@gmail.com> wrote:
Show 60 quoted lines
> From: Lars Schneider <larsxschneider@gmail.com>
>
> Junio usually pushes many commits at once to the public "pu"/"next"/
> "master" branches. If a test fails then it is not obvious what commit
> caused the failure. Therefore we run Git bisect with the merge base
> between the failing rev and its more stable branch ("next" for "pu",
> "master" for "next", and "maint" for "master") as good ref to find the
> offending commit. This is only enabled on "github.com/git/git" because
> there we can assume that all relevant branches are up to date.
>
> Signed-off-by: Lars Schneider <larsxschneider@gmail.com>
> ---
>  .travis.yml       |  2 +-
>  ci/test-report.sh | 65 +++++++++++++++++++++++++++++++++++++++++++++++++++++++
>  2 files changed, 66 insertions(+), 1 deletion(-)
>
> diff --git a/.travis.yml b/.travis.yml
> index 81d2027..922807b 100644
> --- a/.travis.yml
> +++ b/.travis.yml
> @@ -101,7 +101,7 @@ script: make --quiet test
>  after_failure:
>    - >
>      : '<-- Click here to see detailed test output!                                                        ';
> -    ./ci/test-report.sh
> +    ./ci/test-report.sh $TRAVIS_REPO_SLUG $TRAVIS_BRANCH;
>
>  notifications:
>    email: false
> diff --git a/ci/test-report.sh b/ci/test-report.sh
> index d08a999..8f7adad 100755
> --- a/ci/test-report.sh
> +++ b/ci/test-report.sh
> @@ -1,5 +1,12 @@
>  #!/bin/sh
>  #
> +# Print test results and run Git bisect on failed tests.
> +#
> +REPO_ORG_NAME=$1
> +CURRENT_BRANCH_NAME=$2
> +
> +
> +#
>  # Print test results
>  #
>  for TEST_EXIT in t/test-results/*.exit
> @@ -17,3 +24,61 @@ do
>         fi
>  done
>
> +
> +#
> +# Run Git bisect
> +#
> +run_bisect () {
> +       TEST_SCRIPT=$1
> +       BAD_REV=$2
> +       GOOD_RV=$3
> +       TMPDIR=$(mktemp -d -t "ci-report-bisect-XXXXXX" 2>/dev/null)
> +       cat > "$TMPDIR/bisect-run.sh" <<EOF

If you are doing a re-roll, then you could probably fix the style issue. ' cat >"$TMPDIR..." '

Show 48 quoted lines
> +
> +EOF
> +       chmod +x "$TMPDIR/bisect-run.sh"
> +       git bisect start $BAD_REV $GOOD_RV
> +       git bisect run "$TMPDIR/bisect-run.sh"
> +       if test -e ./t/$TEST_SCRIPT.sh && make --jobs=2 >/dev/null 2>&1
> +       then
> +               cd t && ./$TEST_SCRIPT.sh >/dev/null 2>&1
> +       else
> +               # If the test file does not exist or the build fails then tell
> +               # Git bisect to skip the commit.
> +               exit 125
> +       fi
> +       git bisect reset >/dev/null 2>&1
> +}
> +
> +case "$CURRENT_BRANCH_NAME" in
> +       master) STABLE_BRANCH="maint";;
> +       next)   STABLE_BRANCH="master";;
> +       pu)     STABLE_BRANCH="next";;
> +esac
> +
> +if test "$REPO_ORG_NAME" = "git/git" && test -n $STABLE_BRANCH
> +then
> +       BAD_REV=$(git rev-parse HEAD)
> +
> +       # Travis CI clones are shallow. It is possible that the last good revision
> +       # was not fetched, yet. Therefore we need to fetch all commits on the
> +       # stable branch.
> +       git config remote.origin.fetch "+refs/heads/$STABLE_BRANCH:refs/remotes/origin/$STABLE_BRANCH"
> +       git fetch --unshallow --quiet
> +       LAST_GOOD_REV=$(git merge-base $BAD_REV "remotes/origin/$STABLE_BRANCH")
> +
> +       for TEST_EXIT in t/test-results/*.exit
> +       do
> +               if test "$(cat "$TEST_EXIT")" != "0"
> +               then
> +                       TEST="${TEST_EXIT%.exit}"
> +                       TEST_SCRIPT=${TEST#t/test-results/}
> +                       echo "------------------------------------------------------------------------"
> +                       echo "  $(tput setaf 1)${TEST} Bisect$(tput sgr0)"
> +                       echo "------------------------------------------------------------------------"
> +                       run_bisect $TEST_SCRIPT $BAD_REV $LAST_GOOD_REV
> +                       echo ""
> +                       echo ""
> +               fi
> +       done
> +fi

Regards, Pranit Bauva

Lars Schneider· May 23, 2016, 08:36 UTC · re: Pranit Bauva · lore

Re: [PATCH v1 3/3] travis-ci: run Git bisect on failed tests

Show 7 quoted lines
> On 22 May 2016, at 19:21, Pranit Bauva <pranit.bauva@gmail.com> wrote:
> 
> Hey Lars,
> 
> On Sun, May 22, 2016 at 4:30 PM,  <larsxschneider@gmail.com> wrote:
>> From: Lars Schneider <larsxschneider@gmail.com>
>> 
...
Show 13 quoted lines
>> +
>> +#
>> +# Run Git bisect
>> +#
>> +run_bisect () {
>> +       TEST_SCRIPT=$1
>> +       BAD_REV=$2
>> +       GOOD_RV=$3
>> +       TMPDIR=$(mktemp -d -t "ci-report-bisect-XXXXXX" 2>/dev/null)
>> +       cat > "$TMPDIR/bisect-run.sh" <<EOF
> 
> If you are doing a re-roll, then you could probably fix the style issue.
> ' cat >"$TMPDIR..."  '
Good eye :-) I will fix this, too.

Thank you, Lars

Junio C Hamano· May 23, 2016, 01:22 UTC · re: larsxschneider@gmail.com · lore

Re: [PATCH v1 3/3] travis-ci: run Git bisect on failed tests

larsxschneider@gmail.com writes:
Show 7 quoted lines
> Junio usually pushes many commits at once to the public "pu"/"next"/
> "master" branches. If a test fails then it is not obvious what commit
> caused the failure. Therefore we run Git bisect with the merge base
> between the failing rev and its more stable branch ("next" for "pu",
> "master" for "next", and "maint" for "master") as good ref to find the
> offending commit. This is only enabled on "github.com/git/git" because
> there we can assume that all relevant branches are up to date.

Just FYI, 'git log next..pu' is not a very interesting range unless you also use the "--no-merges" option.

Is there a good way to tell the CI that, instead of testing 'pu' (or a specific branch in general), test all new commits that appear on the first-parent chain between 'master'..'pu'?

It would be ideal if CI can do this every time 'master' and 'pu' gets updated.

 - Find all commits that appear on the first-parent chain between
   'master..pu'.
 - For each of them:
   - Test it.  This tests the merge result and can catch breakages
     introduced by mismerging.
   - Also test its second parent if it is a merge.  This tests the
     tip of each topic branch in isolation.  CI gets a bonus point
     if it can remember that the tip of the topic did not move and
     the commit already passed the test to skip it ;-)
 - Optionally, for those topics that failed the "tip of the topic
   branch" test, "bisect master..$topic" to see where it breaks.

The thing is, "bisect" works only when you have a _single_ cause of breakage, but because 'pu' is a pile of unsorted raw material yet to be sifted into dirt and gem, once you see a failure at the tip of 'pu', mechanical 'bisect run' may not be a very useful tool, as you do not know if there is only one breakage you are looking for.

Because the unit of integration we use is a topic branch (not an individual commit on a topic branch), "this topic is broken", and "this topic itself may be good, but merging it breaks 'pu'" are far more interesting and relevant piece of information than "this commit is what broke 'pu'".

← back to recent threads