Re: [RFC/WIP/PATCH v2] gitweb: add test suite with Test::WWW::Mechanize::CGI

From: Jakub Narebski <jnareb@gmail.com>
Date: Jun 16, 2008, 00:40 UTC
Message-ID: <200806160240.18539.jnareb@gmail.com>
In-Reply-To: <4855634C.8060606@gmail.com>

Lea Wiemann wrote:

Show 7 quoted lines

> Jakub Narebski wrote:
> >
> > 1.) Should we put all tests in one file, or should they be split
> 
> I'd suggest we leave it in a single file until test execution time 
> becomes an issue.  Then (when it has become too large) we'll be able to 
> figure out good boundaries along which to split the test suite.

I wanted to split tests mainly not because of performance, but because of making it easier to maintain. Although perhaps single driver-test, and do()-ing or require()-ing sub-files would be enough.

Show 8 quoted lines

> > 2.) What invariants should we test [...]  Checking for example if all items
> >     are listed in a 'tree' view, or if all inner links (#link) are
> >     valid would be a good start... 
> 
> Yup; completeness of item lists is especially relevant for paginated 
> output.  Also check for the presence and validity of links (like 
> "parent" links, etc.), and for the presence of certain elements (like 
> the file modes in the tree view).

For example if "next" (and like) views really lead to next page.

> Also, with a $ENV{LONG_GIT_TEST} variable or so, we could automatically 
> validate all links for each page we're checking -- it takes a long time, 
> but it's still way more efficient than exhaustive spidering of the whole 
> site.

Good idea.  I would examine how it is done in other tests.

> > (by the way, is there some Perl module for RSS, Atom and OPML validation?)
> 
> I can't find anything on Google right now,

I usually search CPAN first, not Google...

> but piping them into external  
> validators might be just as fine.  Also, since those formats are 
> generated using print statements (which is really error-prone for XML 
> formats), I'd say that a good start would be to check for XML validity.

We can use Test::XML / Test::XML::Valid / Test::XML::Simple for being well-formed XML. If RSS / Atom / OPML have good DTD / XML Schema / / Relax-NG schema / Sablotron rules, they could be checked using that from Perl.

Show 11 quoted lines

> > 3.) What invariants you want to test for your caching efforts, e.g.
> >     checking if cached output matches non-cached
> 
> How about this:
> 
> 1. Run the Mechanize tests (and possibly also the existing t9500 tests) 
> *without* caching, recording the URL's and contents of all pages the 
> test suite accesses.
> 
> 2. Get all those URL's again *with* caching (from a cold cache), and 
> assert that the output is identical.

How would you ensure cold cache?

> 3. Get all those URL's again *with* caching (from a warm cache), and 
> assert that the output is identical.

Well, it might be identical, but it also might have "cached output" somewhere in the output.

> Perhaps also assert that no call  
> to the git binary is made (i.e. everything has actually been cached). 
> (Of course we might need options for the production site to not cache 
> certain things, but let's defer this discussion.)

Or at least (if we don't cache everything, and that could be good idea) to check if there are less git binary calls.

-- 
Jakub Narebski
Poland