perl/Git.pm: Writing a test suite

Git.pm doesn't currently have a test suite, and I think it should get one.

Since Git.pm makes calls to the git binary, it would seem sensible to me 
to create mock output for the external calls, like Grit (a Ruby API) 
does it, here: http://github.com/mojombo/grit/tree/master/test/fixtures 
  (Having a complete test repository to test against like Ruby/Git 
(another API) at 
http://repo.or.cz/w/rubygit.git?a=tree;f=tests/files;hb=HEAD is probably 
too brittle and makes it hard to test edge cases.)

I haven't done testing in perl before, so if someone wants to exchange ideas and/or help me get this set up, please post here or message me on IRC (lea_w/lea_1 on #git at Freenode).

-- Lea

2008/5/30 Lea Wiemann <lewiemann@gmail.com>:

Testing is big in the perl world. I suggest to start with reading : http://www.perlfoundation.org/perl5/index.cgi?testing I'm not overloaded with tuits at the moment, but I'll try to review a bit your testing patches when you post them here.

  Hi,

On Fri, May 30, 2008 at 09:43:48AM +0200, Lea Wiemann wrote:

  I think this is bad idea; you should test not just Git.pm
functionality, but equally importantly also its integration with
(current version of) Git. If Git's output/behaviour changes in some
subtle way, you want to see if it breaks Git.pm.

  There are two possible ways, either using the Git test suite
infrastructure, or the Perl test suite infrastructure. I'm ambivalent
about which one to choose, both have their advantages; I wonder what
others' opinion is. I would personally go with the Perl test suite in
this case but can't support it with much more than just a gut feeling.

Petr Baudis wrote:

Good point. So I'll work on test repositories with real calls to Git where possible.

I'd definitely want to use Perl's testing infrastructure, but probably it's sensible to use both Git's and Perl's infrastructure (e.g. call the Git.pm perl tests from the Git test suite shell scripts, and maybe have setup code in the shell scripts). Here are two questions:

1. Is it OK for the Git.pm tests to be called from the Git test scripts 
but generate different-looking output (since the tests are run in Perl)?

2. I might need external test helpers from CPAN at some point.  Is that 
a bad idea because of dependencies, or are we OK with having additional 
dependencies for running the test suite if necessary?

-- Lea

Lea Wiemann <lewiemann@gmail.com> writes:

Cosmetics. As long as they are not disproportionally chatty nor too terse, and correctly catch breakages and stop (if told to with -i) on the first one, I think it is Ok to be different (e.g. no support for colored output).

As long as you use the ones that is widely used in Perl world I do not think it would be a problem.