create new tag
, view all tags

Testing TWiki the Perl way

In the current release, we stumbled over a problem which only manifests itself on Windows (POSIX::tzset not available). Well, one could argue that running TWiki on Windows is a bad idea anyway, or should be done using the TWiki VM. On the other hand, TWikiPersonal works quite fine, and I wouldn't want to abandon it.

Running the test suite on Windows would have revealed the problem, because TimeTests.pm contains the bad call to tzset, so we know that it hasn't been done prior to the release. And here's why: It is way to tedious. TWiki testing requires a lot of environment which isn't easily transferred to Windows. Right now I haven't found a way to get the unit tests running outside an SVN checkout, and I am definitely not going to run a complete checkout on Windows.

On the other hand, Strawberry perl already brings the Perl test tools to Windows, for example the command prove. I do other Perl stuff with prove, so I would love to run prove on a functional TWiki installation, not only on Windows, but also on SVN.

The first shot

How many lines of code does it take at minimum to run the complete TWikiSuite through prove in a SVN checkout? Answer: four.

Create a file .../core/t/TWikiSuite.t:

     use Test::Simple tests => 1;
     chdir "test/unit";
     my $result = `perl ../bin/TestRunner.pl TWikiSuite.pm 2>&1`;
     ok($result =~ /All tests passed \(\d+\)/s, $result);

Then, from the core directory, run:

     prove -vl

The -v switch makes prove log the tests. The -l switch adds ./lib to @INC, exactly what we need when we're in the core directory of a SVN checkout, or in the root of an installation. That sorta works, but isn't very friendly because it runs the whole suite without giving feedback. If all tests pass, that's fine, but if there's a failure, it is too difficult to find out which of the tests failed. With a few more lines, mimicking the collection of tests in today's TWikiSuite.pm, this can be fixed.

Next steps

I plan to implement a reasonably fast way to create a "testable windows installation", leveraging the already existing twikidev collect and prove. In my own environment, I have a CIFS mounted windows drive to which I can collect the installation, so this would be a two-command trick:

  1. From SVN, collect TWiki plus the unit tests to a directory on CIFS
  2. On windows, start prove from the root of this directory.

Would that be interesting for others, i.e. a candidate for the TWiki SVN?

-- Contributors: Harald Jörg - 2015-12-11


Standardization is good.

-- Peter Thoeny - 2015-12-17

As ever so often, the second step turns out to be considerably harder than the first. Up to now I've encountered a challenge: In the current implementation, TWiki's unit test environment needs to be deeply nested with the TWiki runtime.

  • It explicitly requires setlib.cfg - but why? We are running tests, not a TWiki. I accept that the test cases need to get access to the TWiki locations. There are several (too many) mechanisms for that in different areas (setlib.cfg, environment variables TWIKI_LIBS and TWIKI_ROOT), which is messy enough. But the test driver should not depend on code of the test object.
  • It explicitly requires TWiki.pm. In addition to the "why" question from above, and to the huge amount of stuff this imports, this creates another dependency: TWiki.pm requires LocalSite.cfg. So I need a working installation before I can run unit tests? It should definitely be the other way around. Even for a regression test suite, the test driver should be independent of the test object.
  • Importing LocalSite.cfg from a "running" environment creates some randomness on the actual configuration under test. Again and again test cases need to be fixed to make local changes to the configuration (most notably: switch off CryptToken).
In full SVN installations, the usual practice is to pseudo-install UnitTestContrib and TextFixturePlugin, which is ugly enough. There's still some way to go before the TWiki modules (the test ibject) and the test driver can sit on independent locations. I think I'll start committing after a successful run of the (maybe modified) suite under Windows, and on Raspberry.

-- Harald Jörg - 2016-01-08

The next experiment in this area has been motivated by a support question (Comment Plugin: Use emailto-parameter) and a bug report (Bugs:Item7716). I wanted to test the suspicious sub expandEmail, but I did not want to have an email sent for every test case, nor did I want to register users in my SVN TWiki, nor did I want to install SendMailPlugin. So this is how it works:

  1. Create a directory .../trunk/SendMailPlugin/t
  2. Write a CPAN:Test::More conforming test case .../trunk/SendMailPlugin/t/expandEmail.t

To run the test, change directory to .../trunk/SendMailPlugin and say to your computer:

  • prove -lv -I../core/lib - Explanation of the options:
    • -l : shorthand for -Ilib. As in Perl, add the lib directory, in this case, .../trunk/SendMailPlugin/lib, to @INC.
    • -v : verbose. Show the short comment provided for each test case (can be omitted).
    • -I../core/lib : Picks also the core library (plus installed plugins) to @INC.

More variations:

  • prove -lv -I../core/lib t/expandEmail.t - Yes, you can run an individual test file as well.
  • perl -d -Ilib -I../core/lib t/expandEmail.t - If a test case runs inexplicably wrong, just run it through the debugger.

On the plus side,

  • These are what I call unit tests. Running them still uses practically all of TWiki modules, though.
  • I find the invocation, and especially the debugging, a lot easier than with TWiki's Test::Unit stuff. I haven't even figured out yet how to run unit tests for an extension.
On the minus side,
  • To get the tests to run, some TWiki functions need to be simulated by mock functions. For this first step, I just wrote them to the test case, maybe this should be moved to a test library.
  • Writing Test::More stuff is different from writing Test::Unit stuff. For core, or for extensions which already have a Test::Unit suite, the duplication of text fixtures might bring more effort than profit.

I'll commit the test file. It serves to verify that SendMailPlugin had a bug before revision 29909, and that it is fixed now. For me, it is sort of an alpha stage whether using traditional Perl testing is a good idea - maybe we'll find out that it isn't.

-- Harald Jörg - 2016-01-20

As a next step, I've attached four test cases which I am using a lot for the experiments to run TWiki with new Perl versions. In a SVN checkout, create a directory named t under your core, put these files in, and then from core run prove -vl.

  • TWikiSuite.t is a wrapper around TWikiSuite.pm (1186 test cases - this takes some time)
  • compile_all.t compiles everything under core. This needs the Perl module Test::Compile plus a boatload of Perl modules required to compile the stuff we have in core/lib/CPAN/lib.
  • compile_warnings.t tries to capture warnings in the compilation step and has been very helpful to chase the "unescaped { in regex" stuff coming with Perl 5.24. It does not detect warnings which occur only during runtime.
  • search_bugs.t is a minimalistic unit test to underpin the diagnosis for Bugs:Item7823 and Bugs:Item7824.
Later, this should be pushed to SVN... either directly or as a developer-only contrib.

-- Harald Jörg - 2017-10-20

Nice progress Harald!

-- Peter Thoeny - 2017-10-22

Topic attachments
I Attachment History Action Size Date Who Comment
Unknown file formatt TWikiSuite.t r1 manage 1.0 K 2017-10-20 - 11:44 HaraldJoerg Four test suites which can be used with Perl's prove
Unknown file formatt compile_all.t r1 manage 1.2 K 2017-10-20 - 11:44 HaraldJoerg Four test suites which can be used with Perl's prove
Unknown file formatt compile_warnings.t r1 manage 3.5 K 2017-10-20 - 11:44 HaraldJoerg Four test suites which can be used with Perl's prove
Unknown file formatt search_bugs.t r1 manage 2.4 K 2017-10-20 - 11:44 HaraldJoerg Four test suites which can be used with Perl's prove
Edit | Attach | Watch | Print version | History: r7 < r6 < r5 < r4 < r3 | Backlinks | Raw View | Raw edit | More topic actions
Topic revision: r7 - 2017-10-22 - PeterThoeny
  • Learn about TWiki  
  • Download TWiki
This site is powered by the TWiki collaboration platform Powered by Perl Hosted by OICcam.com Ideas, requests, problems regarding TWiki? Send feedback. Ask community in the support forum.
Copyright © 1999-2017 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.