patch, 2 partsSmoke testing for Git

Here's a non-RFC version of the smoke testing for Git series. It's like v2 of the RFC aside from a s/sever/server/ typo noted by Junio:

Ævar Arnfjörð Bjarmason (2):
  tests: Infrastructure for Git smoke testing
  t/README: Document the Smoke testing

 t/Makefile |   35 ++++++++++++++++++++++++++++++-
 t/README   |   66 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
 t/harness  |   21 +++++++++++++++++++
 3 files changed, 121 insertions(+), 1 deletions(-)
 create mode 100755 t/harness

Add the capability to send smoke reports from the Git test suite.

Currently we only notice bugs in the test suite when it's run manually. Bugs in Git that only occur on obscure platforms or setups that the core developers aren't using can thus go unnoticed.

This series aims to change that. With it anyone that's interested in avoiding bitrot in Git can volunteer to run a smoke tester. A smoke tester periodically compiles the latest version of Git, runs the test suite, and submits a report to a central server indicating how the test run went.

A smoke tester might run something like this in cron:

    #!/bin/sh
    cd ~/g/git
    git fetch
    for branch in maint master next pu; do
        git checkout origin/$i &&
        make clean all &&
        cd t &&
        make smoke_report
    done

The smoker might want to compile git with non-default flags, include bisecting functionality or run the tests under valgrind. Doing that is outside the scope of this patch, this just adds a report submission mechanism. But including a canonical smoke runner is something we'll want to include eventually.

What this does now is add smoke and smoke_report targets to t/Makefile (this example only uses a few tests for demonstration):

    $ make clean smoke
    rm -f -r 'trash directory'.* test-results
    rm -f t????/cvsroot/CVSROOT/?*
    rm -f -r valgrind/bin
    rm -f .prove
    perl ./harness --git-version="1.7.2.1.173.gc9b40" \
                    --no-verbose \
                    --archive="test-results/git-smoke.tar.gz" \
                    t0000-basic.sh t0001-init.sh t0002-gitfile.sh t0003-attributes.sh t0004-unwritable.sh t0005-signals.sh t0006-date.sh
    t0000-basic.sh ....... ok
    t0001-init.sh ........ ok
    t0002-gitfile.sh ..... ok
    t0003-attributes.sh .. ok
    t0004-unwritable.sh .. ok
    t0005-signals.sh ..... ok
    t0006-date.sh ........ ok
    All tests successful.

    Test Summary Report
    -------------------
    t0000-basic.sh     (Wstat: 0 Tests: 46 Failed: 0)
      TODO passed:   5
    Files=7, Tests=134,  3 wallclock secs ( 0.06 usr  0.05 sys +  0.23 cusr  1.33 csys =  1.67 CPU)
    Result: PASS

    TAP Archive created at /home/avar/g/git/t/test-results/git-smoke.tar.gz

The smoke target uses TAP::Harness::Archive to aggregate the test results into a tarball. The tarball contains two things, the output of every test file that was run, and a metadata file:

Tarball contents:

    $ tar xzvf git-smoke.tar.gz
    t0004-unwritable.sh
    t0001-init.sh
    t0002-gitfile.sh
    t0005-signals.sh
    t0000-basic.sh
    t0003-attributes.sh
    t0006-date.sh
    meta.yml

A test report:

    $ cat t0005-signals.sh
    ok 1 - sigchain works
    # passed all 1 test(s)
    1..1

A metadata file:

    ---
    extra_properties:
    file_attributes:
      -
        description: t0000-basic.sh
        end_time: 1280437324.61398
        start_time: 1280437324.22186
      -
        description: t0001-init.sh
        end_time: 1280437325.12346
        start_time: 1280437324.62393
      -
        description: t0002-gitfile.sh
        end_time: 1280437325.29428
        start_time: 1280437325.13646
      -
        description: t0003-attributes.sh
        end_time: 1280437325.59678
        start_time: 1280437325.30565
      -
        description: t0004-unwritable.sh
        end_time: 1280437325.77376
        start_time: 1280437325.61003
      -
        description: t0005-signals.sh
        end_time: 1280437325.85426
        start_time: 1280437325.78727
      -
        description: t0006-date.sh
        end_time: 1280437326.2362
        start_time: 1280437325.86768
    file_order:
      - t0000-basic.sh
      - t0001-init.sh
      - t0002-gitfile.sh
      - t0003-attributes.sh
      - t0004-unwritable.sh
      - t0005-signals.sh
      - t0006-date.sh
    start_time: 1280437324
    stop_time: 1280437326

The "extra_properties" hash is where we'll stick Git-specific info, like whether Git was compiled with gettext or the fallback regex engine, and what branch we're compiling. Currently no metadata like this is included.

The entire tarball is then submitted to a central smokebox at smoke.git.nix.is. This is done with curl(1) via the "smoke_report" target:

    $ make smoke_report
    curl \
                    -H "Expect: " \
                    -F project=Git \
                    -F architecture=x86_64 \
                    -F platform=Linux \
                    -F revision="1.7.2.1.173.gc9b40" \
                    -F report_file=@test-results/git-smoke.tar.gz \
                    http://smoke.git.nix.is/app/projects/process_add_report/1 \
            | grep -v ^Redirecting
      % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                     Dload  Upload   Total   Spent    Left  Speed
    100  117k  100    63  100  117k      3   6430  0:00:21  0:00:18  0:00:03     0
    Reported #8 added.

Reports are then made available on the smokebox via a web interface:

    http://smoke.git.nix.is/app/projects/smoke_reports/1

The smoke reports are also mirrored to a Git repository hosted on GitHub:

    http://github.com/gitsmoke/smoke-reports

The Smolder SQLite database that contains metadata about the reports is also made available:

    http://github.com/gitsmoke/smoke-database

Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>
---
 t/Makefile |   35 ++++++++++++++++++++++++++++++++++-
 t/harness  |   21 +++++++++++++++++++++
 2 files changed, 55 insertions(+), 1 deletions(-)
 create mode 100755 t/harness

On Mon, Aug 2, 2010 at 4:24 PM, Ævar Arnfjörð Bjarmason <avarab@gmail.com> wrote:

You might want to play with gitbuilder: http://github.com/apenwarr/gitbuilder

It can bisect automatically and it looks like it could integrate pretty nicely with this system. (If you're running a gitbuilder, just have it submit the smoke test with each build it makes. gitbuilder will automatically bisect if 'make test' fails, so eventually you'll submit smoke tests that *don't* fail.)

Have fun,

Avery

Git now has a smoke testing service at http://smoke.git.nix.is that anyone can send reports to. Change the t/README file to mention this.

Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>
---
 t/README |   66 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
 1 files changed, 66 insertions(+), 0 deletions(-)

Heya,

On Mon, Aug 2, 2010 at 15:24, Ævar Arnfjörð Bjarmason <avarab@gmail.com> wrote:

Can't we run this somewhere a little less "shabby"? I'd hate to introduce Yet Another Official Git Site (I'm really glad for example that the Git Wiki is now on kernel.org). Is it not possible to move this to git-scm.com, kernel.org, or something like that?

On 08/02/2010 04:54 PM, Sverre Rabbelier wrote:

And what's on the site tells us almost nothing about the tested systems.

On Mon, Aug 2, 2010 at 21:04, A Large Angry SCM <gitzilla@gmail.com> wrote:

Do you not like the domain, that the domain's A record points to a Linode box, or both?

I don't mind where a smoke server gets hosted, I just want it to be there. But someone's going to have to host it and maintain it.

Unless someone else wants to do that it's going to have to either be at a box I control, or someone else will need to give me shell access.

Of course just changing the domain would be trivial, just a matter of someone setting up an A record somewhere, or delegating a zone to me.

An alternative would be to change how this works, e.g. making submissions go by E-Mail to a public list, or having smokers push them to public Git repositories which would be registered somewhere.

But that'd require more development and setup work. The current system isn't optimal, but it works. It can always me moved somewhere else later if smoke testing for Git takes off.

Can you be more specific? What information do you find lacking at http://smoke.git.nix.is/app/projects/smoke_reports/1 ?

Heya,

[+Warthog9]

On Mon, Aug 2, 2010 at 17:00, Ævar Arnfjörð Bjarmason <avarab@gmail.com> wrote:

The former.

Agreed.

John, is it possible to have smoke.git.kernel.org (or such a domain) either point at a server Ævar controls, or to give him a shell account there? The point is to have http://smoke.git.nix.is/ hosted on a more "Gitty" domain.

I like the idea of having it be a public Git repo since that would probably make it easier to do maintenance on the repo (e.g., revert "vandalism"), but there still needs to be a frontend of sorts somewhere :).

On Mon, Aug 2, 2010 at 22:05, Sverre Rabbelier <srabbelier@gmail.com> wrote:

Note that that may be a bit premature, since Junio hasn't commented on whether he wants this at all yet.

Note that all the data is already in Git[1] it just isn't used as the submission mechanism.

1. http://github.com/gitsmoke

Ævar Arnfjörð Bjarmason wrote:

Operating system (i.e., distro). Precise CPU architecture. Number of CPUs. Available memory. Which tests were skipped and which had expected failures. Configuration used to build git.

(I understand that requiring all that information could prevent people from running smoke servers, so I don’t mean to say it all should be required.)

On Mon, Aug 2, 2010 at 22:13, Jonathan Nieder <jrnieder@gmail.com> wrote:

That'd be nice. We could detect that and include it later in extra_properties in t/harness. It'd need some fancy detection code though.

That's actually already there.

On 08/02/2010 06:00 PM, Ævar Arnfjörð Bjarmason wrote:

[...]

The only information available there is that the tested system was some Linux on an x86_64 and the particular git version tested. My complaint is that "Linux" is _way_ too generic to be useful when trying to get an idea of where things are known to work and where they are known not to work.

Ævar Arnfjörð Bjarmason <avarab@gmail.com> writes:

"shabby" is not a good word to describe a service somebody volunteered to maintain for the public good. Be civil, please.

"Platform Linux" does not tell much about the tested systems, no?

Heya,

On Mon, Aug 2, 2010 at 17:44, Junio C Hamano <gitster@pobox.com> wrote:

That's why I quoted it, I didn't mean to put Ævar's work down and I think he understood what I meant, judging from his reply. On that note, thanks Ævar for reading what I meant, not what I said :).