From: Eric Sunshine <sunshine@sunshineco.com>
Date: Tue, 04 Sep 2018 00:43:34 GMT
Subject: Re: [PATCH 4/9] tests: optionally write results as JUnit-style .xml
Message-ID: <CAPig+cTJzguROwyGgyeUt5CrpKt2kyps04QD37wa_Bt3JPwWsg@mail.gmail.com>
In-Reply-To: <2b5d6785944ed1a66f0272896a45fece8521d10f.1536009027.git.gitgitgadget@gmail.com>

On Mon, Sep 3, 2018 at 5:10 PM Johannes Schindelin via GitGitGadget
<gitgitgadget@gmail.com> wrote:
> This will come in handy when publishing the results of Git's test suite
> during an automated VSTS CI run.
>
> Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
> ---
> diff --git a/t/test-lib.sh b/t/test-lib.sh
> @@ -431,11 +434,24 @@ trap 'exit $?' INT
>  test_failure_ () {
> +       if test -n "$write_junit_xml"
> +       then
> +               junit_insert="<failure message=\"not ok $test_count -"
> +               junit_insert="$junit_insert $(xml_attr_encode "$1")\">"
> +               junit_insert="$junit_insert $(xml_attr_encode \
> +                       "$(printf '%s\n' "$@" | sed 1d)")"
> +               junit_insert="$junit_insert</failure>"

This is a genuine failure, so you're creating a <failure> node. Okay.

> +               write_junit_xml_testcase "$1" "      $junit_insert"
> +       fi
> @@ -444,11 +460,19 @@ test_failure_ () {
>  test_known_broken_ok_ () {
> +       if test -n "$write_junit_xml"
> +       then
> +               write_junit_xml_testcase "$* (breakage fixed)"
> +       fi
>         test_fixed=$(($test_fixed+1))
>         say_color error "ok $test_count - $@ # TODO known breakage vanished"
>  }

This was expected to fail but didn't, which means it probably needs
some sort of attention. test_known_broken_ok_() prints this result in
the 'error' color, and test_done() re-inforces that by printing a
message, also in 'error' color:

    42 known breakage(s) vanished; please update test(s)

So, should this emit a <failure> node also, perhaps with 'type'
attribute set to "warning" or something? (<failure type="WARNING"
message="...">)

> @@ -758,9 +793,58 @@ test_at_end_hook_ () {
> +xml_attr_encode () {
> +       # We do not translate CR to &#x0d; because BSD sed does not handle
> +       # \r in the regex. In practice, the output should not even have any
> +       # carriage returns.
> +       printf '%s\n' "$@" |
> +       sed -e 's/&/\&amp;/g' -e "s/'/\&apos;/g" -e 's/"/\&quot;/g' \
> +               -e 's/</\&lt;/g' -e 's/>/\&gt;/g' \
> +               -e 's/  /\&#x09;/g' -e 's/$/\&#x0a;/' -e '$s/&#x0a;$//' |
> +       tr -d '\012\015'
> +}

It's possible to insert a literal CR in the 'sed' expression, which
does match correctly on BSD (and MacOS). For instance:

    CR=$(printf "\r")
    sed -e "s/$CR/\&#x0d;/g"