Re: [RFC/WIP/PATCH v2] gitweb: add test suite with Test::WWW::Mechanize::CGI
- From
- Lea Wiemann <lewiemann@gmail.com>
- Date
- Jun 15, 2008, 18:45 UTC
- Message-ID
- <4855634C.8060606@gmail.com>
- In-Reply-To
- <200806152001.34350.jnareb@gmail.com>
Jakub Narebski wrote:
> 1.) Should we put all tests in one file, or should they be split
I'd suggest we leave it in a single file until test execution time becomes an issue. Then (when it has become too large) we'll be able to figure out good boundaries along which to split the test suite.
> 2.) What invariants should we test [...] Checking for example if all items > are listed in a 'tree' view, or if all inner links (#link) are > valid would be a good start...
Yup; completeness of item lists is especially relevant for paginated output. Also check for the presence and validity of links (like "parent" links, etc.), and for the presence of certain elements (like the file modes in the tree view).
Also, with a $ENV{LONG_GIT_TEST} variable or so, we could automatically validate all links for each page we're checking -- it takes a long time, but it's still way more efficient than exhaustive spidering of the whole site.
> (by the way, is there some Perl module for RSS, Atom and OPML validation?)
I can't find anything on Google right now, but piping them into external validators might be just as fine. Also, since those formats are generated using print statements (which is really error-prone for XML formats), I'd say that a good start would be to check for XML validity.
> 3.) What invariants you want to test for your caching efforts, e.g. > checking if cached output matches non-cached
How about this:
1. Run the Mechanize tests (and possibly also the existing t9500 tests) *without* caching, recording the URL's and contents of all pages the test suite accesses.
2. Get all those URL's again *with* caching (from a cold cache), and assert that the output is identical.
3. Get all those URL's again *with* caching (from a warm cache), and assert that the output is identical. Perhaps also assert that no call to the git binary is made (i.e. everything has actually been cached). (Of course we might need options for the production site to not cache certain things, but let's defer this discussion.)
-- Lea