TWiki Test Infrastructure

TBD

Contributors:
-- PeterThoeny - 02 Nov 2003

Discussions

TWiki needs a test infrastructure. This helps in making the barrier to contribution lower, thus accelerating the TWiki development. This is because contributors can test their code changes before submitting a patch, and the core team has an easier job of accepting patches. Currently it is very time consuming for the core team to manually test a patch to make sure that it does not break existing content. I spend probably 80% of my time on making sure that there are no issues with existing content.

There are several aspects to testing, like quality of code, efficiency of code and compatibility of code changes. The latter one has high priority since the TWikiMission states that topic content should be protected from data corruption and incompatible changes.

The TWiki test infrastructure should be automated for core code and Plugins, and should focus on compatibility of code changes.

Idea for a TWiki-based automated testing:

• Create a TWikiTestInfraPlugin
• Create a test topic for each test area like TWikiVariablesTest, FormattedSearchTest, LinkTest, etc
• Each test topic lists test cases (TC) of a certain format. A table format and a free format should be supported.
• Test cases in table format example:

  Result	Input	Expected	Comment
  %TC%	*bold*	bold	Bold text
  %TC%	(*bold*)	(bold)	Bold in parens
  %TC%	*not bold*	not bold	Not bold because of leading > sign

• Test cases in free format are needed to test rendering outside of tables, like at the beginning of a line, in a bullet etc.
• The Plugin runs the test and expands the %TC% with the Pass or Fail result. The results can be summarized with the SpreadSheetPlugin. All results can be carried over into a overall result page.

-- PeterThoeny - 02 Nov 2003

please remember UnitTestSuite

from a very hazy memory - i think that we should consider seperating each bit of info into seperate topics.

a UnitTestTopic which is the configuration file that specifies an ExpectedUnitTestResultTopic (and there might be a good argumant for having the UnitTestinputTopic seperate too.

-- SvenDowideit - 03 Nov 2003

The "testcases" approach described above is of course very valid. However my experience tells me that while this approach would work, it is high maintenance and at high risk of frustrating plugin authors.

To illustrate what I mean, let's say a plugin is defined that post-filters a page and converts all "http:" links to a particular URL to "https:". The test pages you have defined suddenly become useless to the plugin author, because they are still expecting "http:". They can't realistically derive a new set of test pages, because interactions with other plugins become way too complex to handle.

Such testcases are very fragile; small changes in the core code can invalidate the whole test suite. That's what I mean by high maintenance.

The testcases approach is probably necessary, but definitely not sufficient.

The right approach is of course unit testing. However reverse engineering a unit test suite onto an existing mature codebase is a huge challenge. The main reason is (again from experience) that as you write the tests, you find the holes in the CUT (code under test) and have to fix them before sensible progress can be made. The process of writing tests is itself a test process.

The second reason reverse engineering tests can be difficult is because the CUT has become intimately threaded; every piece depends on every other piece, and isolating functionality for test is impossible without significant rewriting. If the rewriting is not done, then the tests themselves are impossible to write and ipso facto the CUT doesn't get fully tested.

This suggests that unit test development is a driver for structural re-engineering. Instead of saying "let's re-engineer the code base to make it more modular", let's say "let's write a full suite of unit tests for each interface". The goal becomes not "modular wiki" but "a fully automated set of unit tests for each interface in the system". The process of writing the tests drives the necessary refactoring.

BTW I'd be quite happy to put my money where my mouth is on this. I'm test-infected; I like writing tests. The only reason I haven't to date is that writing those tests implies the re-engineering of the core suggested above, and would generate a huge volume of changes that would probably scare the core team to death.....

-- CrawfordCurrie - 29 Nov 2003

My primary interest at this time is accelerated Core development. One roadblock is the effort it takes to verify that the change does not break existing content.

Unit testing is important but a "nice to have" compared to the verification of code in regards to properly rendering content.

Test cases do not mean high maintenace if designed properly. That is, the problem Crawford mentions can be solved if Plugin are able to overload test cases.

One additional advantage of the test case approach is that the underlining code can be changed without changing the test cases, e.g. when modularizing the code base.

-- PeterThoeny - 14 Dec 2003

I understand and sympathise with your goals. I don't necessarily agree with the test cases .vs. unit tests view, but that's my problem. I will continue to write both.

Modern practice (test first programming) means that the tests lead the development. So, if you are going to modularise code, the first thing you do is write the tests for the new module. Then, you write/refactor code until the tests pass.

It's an interesting intellectual challenge to work out a good, clean way for plugins to overload test cases.

-- CrawfordCurrie - 14 Dec 2003

The scenario you describe makes sense for new projects. At the stage where TWiki is, it is a very expensive undertaking to refactor code for unit testing. It can be introduced slowly at the time of code cleanup like when modularizing the code base, but I believe the cost/benefit ratio does not justify code cleanup for the sake of unit testing.

-- PeterThoeny - 14 Dec 2003

That's the recommended practice when retrofitting code with unit test: Add tests to the code you'll be touching for the behavior you want to guarantee.

Later, if something goes wrong in an unsuspecting place, write the test to verify the error. This way the test harness is build step by step.

On another topic, there is a framework for acceptance testing called FIT (created by Ward Cunningham) that was translated to Perl and is available at http://fit.c2.com/wiki.cgi?PerlDownloads that looks like what Peter described.

-- RafaelAlvarez - 05 Jan 2004

Note that KWiki are working on a FIT module generator. http://www.kwiki.org/index.cgi?KwikiFit

-- CrawfordCurrie - 24 Feb 2004

One idea I had for testing changes to the TWiki core code was to have a standard TWiki test suite web. The idea is that you take an entire web's content, and write some method for generating "canonical" TWiki output for every page. This output is then saved as HTML in a parallel folder. There should be some way of saving and recovering the exact settings used to produce this output.

Then, after you change TWiki, you rerun the generation to a new target folder, and compare the results. Any differences will be spotted, and it is up to the developer to decide whether the changes are acceptable or not.

-- WalterMundt - 24 Feb 2004

When I was making the changes for TWiki/UI I wrote a test suite using Test::Unit that compares a "golden" twiki installation against a "test" installation using the same data. I think the code is all checked in under 'test'.

-- CrawfordCurrie - 25 Jul 2004

If we have enough developers actually doing work for DakarRelease, I will undertake to look after this - unfortuanatly with Cairo it has been impossible for me to find time..

so Dakar&Me this topic is.

-- SvenDowideit - 25 Jul 2004

And I will work on the test case infrastructure. This will allow black box testing of large scale code refactor that do not have unit tests.

-- PeterThoeny - 25 Jul 2004

Crawford did KickStartingTestcaseDefinitions. This is a step in the right direction, exactly what I was asking for one year ago. This will be a big relief for everyone: The contributors who can test their code against, and the core team who can accept code quicker based on the test results. It will be an initial effort to document the test cases, but this is worth the time since we will gain later on.

The TWiki Test Infrastructure should be documented on top of this page once we defined the "how to".

Requirements, terminology and high level spec as I see it:

• Use a new web for test cases. Name it TWTC, TWikiTC, TWikitest?
• A TestSet is a topic that define several TestCases
• Each TestCase is defined as follows:
  • a section that contains the test input
  • a section that contains the expected output
  • a test result indication (failed/passed/skipped)
  • a comment
• Support two types of test case definitions:
  • A simple type in table format, as described above on 02 Nov 2003
  • A flexible type that may span several lines, as proposed by Crawford in KickStartingTestcaseDefinition
• View a TestSet topic to see result of all TestCases, with statistics (# passed, # failed, # skipped).
  • Should a normal topic view run the tests directly, or should it be enabled with a switch, such as a twtc=run parameter? (which could be a link on the TestSet topic)
• Once a test is run successfully, its test result may be recorded on the TopicSet topic (press a button to update the topic's TWikiForm with the result, which would be initiated a twtc=record parameter).
  • Results of older test runs can be viewed in the topic history.
• An index topic pulls a set of TestSet topics together and shows an overall test result summary. This can be a simple SEARCH table, pulling the test results from the forms, with a SpreadSheetPlugin summary row at the end
  • The index topic may also have a button to test and record all TestSets

-- PeterThoeny - 22 Oct 2004

We need to account also for Plugin testing. As proposed in KickStartingTestcaseDefinition, a new web for test cases seems appropriate. How can this be combined with Plugin testing? For KISS, it might be better to keep the Plugin tests in the same web where the Plugin is shipped, e.g. the TWiki web.

For example the (unpublished) SpreadSheetPlugin tests are done manually, the EVEN function is simply a manual inspection of:

• $EVEN(-2) = 1
• $EVEN(-1) = 0
• $EVEN(0) = 1
• $EVEN(1) = 0
• $EVEN(1.5) = 0
• $EVEN(2) = 1
• $EVEN(3) = 0

Each Plugin could ship with a ...PluginTestCases topic, e.g. SpreadSheetPluginTestCases. Above example would look like this:

   * $EVEN() tests
     | *Result* | *Input* | *Expected* | *Comment* |
     | %TC% | %CALC{"$EVEN(-2)"}% | 1 | $EVEN(-2) |
     | %TC% | %CALC{"$EVEN(-1)"}% | 0 | $EVEN(-1) |
     | %TC% | %CALC{"$EVEN(0)"}% | 1 | $EVEN(0) |
     | %TC% | %CALC{"$EVEN(1)"}% | 0 | $EVEN(1) |
     | %TC% | %CALC{"$EVEN(1.5)"}% | 0 | $EVEN(1.5) |
     | %TC% | %CALC{"$EVEN(2)"}% | 1 | $EVEN(2) |
     | %TC% | %CALC{"$EVEN(3)"}% | 0 | $EVEN(3)|

which gets expanded by the TWikiTestInfraPlugin into:

• $EVEN() tests

  Result	Input	Expected	Comment
  pass	1	1	$EVEN(-2)
  pass	0	0	$EVEN(-1)
  pass	1	1	$EVEN(0)
  pass	0	0	$EVEN(1)
  pass	0	0	$EVEN(1.5)
  pass	1	1	$EVEN(2)
  pass	0	0	$EVEN(3)

-- PeterThoeny - 25 Oct 2004

No problem with the idea of having test cases for plugins. Not sure if you are proposing shipping them, which I'm OK with as long as they are shipped as part of a zipped test suite. Not everyone wants or needs these testcases. Plugins built with BuildContrib already ship their unit tests this way.

I assume you are proposing that the test plugin implements an endRenderingHandler, which processes the rendered result and compares expected and actual.

There are a few problems I can see with this approach.

1. Because plugins are invoked together in the middle of the rendering pipe, there is no way to isolate the results of the processing done by a single plugin. Results may be (1) processed further by another plugin or (2) munged by the rest of the rendering loop. And because different plugins may be installed, the results might be different.
2. There is no easy way to express expected results, such that a test plugin can use them. If they are expressed in plain in a table as suggested above, then those results are subject to the same rendering processes as anything else. Enclosing them in verbatim tags is not enough, as even varbatim receives some processing. Even HTML comments get attacked in the rendering loop. The only way is for the test plugin to re-read the test topic raw, and then you have the problem of matching up the "expected" against the corresponding "actual".
3. It requires you to parse HTML (CPAN:HTML::TableExtract could useful here)
4. While this approach gives a way to test rendered output, there is no obvious way to set up fixtures (for example, to adjust the permissions in the web). Fixtures are needed for testing most plugins.

-- CrawfordCurrie - 25 Oct 2004

I implemented the TestFixturePlugin in DEVELOP, version 1814. Also updated the testcases, version 1815.

-- CrawfordCurrie - 04 Nov 2004

Along with the unit test infrastructure, this is ready to rock on the develop branch. The TestFixturePlugin (not in plugins web, in TWiki) does pretty much what is described above. There is also an extensive set of testcases, both automated and manual. Note that unfortunately, MAIN and Cairo cannot run the testcases due to recursion bugs in Search.

-- CrawfordCurrie - 11 Jan 2005

Code Coverage Analysis in Perl

Creating Perl Code Graphs

-- WillNorris - 27 Apr 2005

Removed from Categorytesting, because it is obsolete.

-- CrawfordCurrie - 02 Dec 2006