Re: [PATCH] travis-ci: include the trash directories of failed tests in the trace log

From: SZEDER Gábor <szeder.dev@gmail.com>
Date: Aug 7, 2018, 18:58 UTC
Message-ID: <CAM0VKjkdf7Rn7nxZQ5DScT_nYAbK76SFuWRUrcUJb23utKWf7A@mail.gmail.com>
In-Reply-To: <51083CBF-E805-4408-8973-E5C85100E319@gmail.com>

On Tue, Aug 7, 2018 at 5:12 PM Lars Schneider <larsxschneider@gmail.com> wrote:

Show 17 quoted lines

>
> > On Aug 1, 2018, at 12:56 AM, SZEDER Gábor <szeder.dev@gmail.com> wrote:
> >
> > The trash directory of a failed test might contain invaluable
> > information about the cause of the failure, but we have no access to
> > the trash directories of Travis CI build jobs.  The only feedback we
> > get from there is the build job's trace log, so...
>
> 100% agree that keeping the trash directory is valuable.
>
> > Modify 'ci/print-test-failures.sh' to create a tar.gz archive of the
> > trash directory of each failed test, encode that archive with base64,
> > and print the resulting block of ASCII text, so it gets embedded in
> > the trace log.  Furthermore, run tests with '--immediate' to
> > faithfully preserve the failed state.
>
> I feel that might be complicated to work with.

It works quite conveniently in practice, that was one of the points of the next paragraph.

> How about this:
> We create an publicly available Git repo (e.g. github.com/git/git-test-trash`)

I'm not sure I understand. A publicly _writable_ repository? That sounds like asking for trouble.

Show 5 quoted lines

> and then we push the trash directory of every failed test run to
> this repo. Then we link the commit in the logs to help people find
> them.
>
> Wouldn't that make it easier to access the trash dir?

With this patch, if there are any trash directories embedded in the log, then the log will show a command near the very end that you can copy-paste into a terminal, and bam! in less than 3 seconds you have all those trash directories downloaded and unpacked.

I'm not sure how it could be made any easier than that.

Show 19 quoted lines

> > Extracting the trash directories from the trace log turned out to be a
> > bit of a hassle, partly because of the size of these logs (usually
> > resulting in several hundreds or even thousands of lines of
> > base64-encoded text), and partly because these logs have CRLF, CRCRLF
> > and occasionally even CRCRCRLF line endings, which cause 'base64 -d'
> > from coreutils to complain about "invalid input".  For convenience add
> > a small script 'ci/util/extract-trash-dirs.sh', which will extract and
> > unpack all base64-encoded trash directories embedded in the log fed to
> > its standard input, and include an example command to be copy-pasted
> > into a terminal to do it all at the end of the failure report.
> >
> > A few of our tests create sizeable trash directories, so limit the
> > size of each included base64-encoded block, let's say, to 1MB.  And
> > just in case something fundamental gets broken and a lot of tests fail
> > at once, don't include trash directories when the combined size of the
> > included base64-encoded blocks would exceed 1MB.
> >
> > Signed-off-by: SZEDER Gábor <szeder.dev@gmail.com>
> > ---