Feedback on CliRunnerContrib

A Contrib to run TWiki from the command line

Yes, of course, everyone can do that.

Just chdir to your bin directory and say view Main.WebHome to your computer.

So what's the point?

Apparently, this contribution is not targeted at normal TWiki installation. On the other hand, it has some nice use cases for TWiki developers. I have encountered some of them recently, enough for me to start hacking it into a contrib:

1. This one I came across when thinking about BenchmarkFramework and fiddling with BenchmarkContrib (only in SVN): Wouldn't it be nice if I could change some configuration item, to check its effect on performance, without having to fiddle with LocalSite.cfg?

       my $runner  =  TWiki::Contrib::CliRunnerContrib->new();
   
       $runner->twikiCfg({UserInterfaceInternationalisation => 1});
       # start taking time
       my $result  =  $runner->run();
       # stop taking time
   
       $runner->twikiCfg({UserInterfaceInternationalisation => 0});
       # start taking time
       my $result  =  $runner->run();
       # stop taking time
       

2. An idea from TestingFromTheCommandLine: Wouldn't it be nice if the so-called "automatic" tests from SVN's TestCases web could really run automatically in TWiki's unit test framework?

       sub test_AutoAttributeParsing {
           my $this = shift;
           my $runner  =  TWiki::Contrib::CliRunnerContrib->new();
           $runner->twikiCfg([qw(Plugins TestFixturePlugin Enabled)] =>1});
           $runner->topic('TestCases.TestCaseAutoAttributeParsing');
           my $result  =  $runner->run();
           $this->assert_matches(qr/ALL TESTS PASSED/,$result);
       }
       

3. Inspired by the checkers in the TWiki4.1 rewrite of bin/configure: Suppose I write a piece of code which depends on some CPAN module, e.g. CGI::Session. I have this module installed in its standard path /usr/lib/perl5/site_perl/...., of course. But would everyone else? I can wrap the module usage in eval. But wouldn't it be nice to verify in a testcase that my plugin produces an intellegible error message if the module isn't there?

       my $runner  =  TWiki::Contrib::CliRunnerContrib->new(script => 'configure');
       $runner->no('CGI::Session');
       my $result  =  $runner->run();
       

I am currently trying to write this stuff, fighting with some rough edges (e.g. how to choose between "replace" and "merge" in multi-level configuration items), and with documentation, and last not least with testcases which are more a showcase for this contrib.

If you're interested in using such a contrib: comments, ideas, and criticism is welcome.

-- HaraldJoerg - 09 Nov 2006

Interesting idea. Doesn't it overlap rather a lot with TWikiShellContrib?

On point 1, excellent idea; but wouldn't it be easier just to write a unit test to do that? Or if that's too complex, why not just time perl view ... ?

On on point 2, not sure why you wouldn't just port the testcases web into unit testcases, as it gives you a lot more flexibility in the testing process. I am extremely discouraged by the fact that after I was pushed to support testcases in TWiki topics, and putting a huge amount of personal effort into the infrastructure, there has been bugger all support for writing the things.

On point 3; not sure what an "intelligible error message" means. When I first started out with perl, I recall having a terrible time resolving module locations because the error messages perl produces by default are so obscure.

I like the idea of making writing tests easier for plugin authors, though. Perhaps this is a way to make life easier for them. Not that I have any faith in their will or ability to write tests, though

-- CrawfordCurrie - 11 Nov 2006

On the overlap with TWikiShellContrib: Maybe it overlaps. I have to admit that I simply don't understand TWikiShellContrib. For example, I can't see how it would help with any of my use cases, or what I would have to do to make it helpful. From the examples I saw (after installing in SVN - the topic at twiki.org is incomplete) I'd rather say TWikiShellContrib overlap with create-new-plugin.pl and Kenneth's apache config generator.

Performance measurements within the unit test framework are, in my limited experience, of very limited use. In a unit test, many TWiki modules have been already compiled in advance, by the test framework. And if they haven't, then the first test case gets the penalty for compilation but the second doesn't. Since tests are picked at random, you don't even know in advance which one will be run first. So if I have to put the things I want to compare in separate files anyway, and run them separately, then I doubt that that would still be easier than using the Contrib.

In the unit tests I can time individual routines, but I can not profile with e.g. Devel::DProf, which would add up runtime of the target routine with runtime of the test case framework runtime.

The approach I have in mind is of course similar to time perl view, but extends it by having the opportunity to change configuration items for one single run of view, without needing to change and restore LocalSite.cfg. Or even to exchange a TWiki module, without overwriting it in the lib directory, like in (not yet implemented)

   $runner->use('TWiki::Render' => '~/Mylib/FastRender.pm');

On point 2, as in TestingFromTheCommandLine, I fully agree with you. Unit tests may be the proper way to do it. On the other hand, using the Contrib to simply view all topics matching TestCaseAuto and assert that the result contains ALL TESTS PASSED looks pretty simple, compared to clicking all these - especially if you want to do it regularly. I think I may be able to commit a TWiki unit test case which simply happens to use the topics in the TestCases web.

On point 3: TWiki does many things to avoid printing "TWiki detected an internal error" whereever possible, like for example the said wrapping of module usage in eval blocks. I'd just think it would be cute to have a way to verify that, if the module isn't present, the result of the eval failure does something more intelligent than printing "TWiki detected an internal error". Call it "testing graceful degradation".

Finally, I think that the Contrib is even more useful for writing core than plugin tests. I hope that the technique will allow to test your examples from TestingFromTheCommandLine, like a complete registration cycle, easier than it is today. Just wrap a couple of run calls into one unit test. I know that I'll have to prove that by giving some convincing examples

-- HaraldJoerg - 11 Nov 2006

Anything that makes testing easier (and preferably more automated) gets my vote!

-- CrawfordCurrie - 12 Nov 2006

The shell contrib in itself just provides a command line shell environment where you can initialise TWiki and up-arrow to recall previous commands, etc.

Raf built a CommandSet for the shell contrib that adds some of the testing you describe. You'd have to ask him about that.

-- MartinCleaver - 12 Nov 2006

With SVN r11967 and r11969 I have committed a first shot at the Contrib and at a first use case (the TestCases web thing). If I eventually upload to twiki.org, then the current topic will be renamed to ...Dev, because the build process auto-creates a documentation topic.

-- HaraldJoerg - 13 Nov 2006

With anticipation to the release I renamed this topic from CliRunnerContrib to CliRunnerContribDev.

Thanks Harald for working on this and for sharing it with the community!

-- PeterThoeny - 13 Nov 2006

Thanks Peter for moving the topic I've uploaded the SVN contents now. And there's another use case: test/unit/ViewParamSectionTests.pm is using the Contrib's ability to capture STDOUT - I've no idea how to do that otherwise.

-- HaraldJoerg - 13 Nov 2006

Thanks Harald for posting the first release.

• I changed the attached form, please fill in the details.
• I escaped the linking of CliRunnerContrib in the heading.
• I tagged the contrib topic so that it can be found easily. Help with the tag votes.

-- PeterThoeny - 14 Nov 2006

Thanks! I wonder whether BuildContrib will retain the form when I run a new upload? Well, we'll see

-- HaraldJoerg - 14 Nov 2006

Harald, some feedback on latest contrib page:

• Suggest to escape linking to oneself (see heading)
• "This Contrib requires the yet-to-be released TWiki version 4.1!" is outdated
• Contrib version number is 0, possibly a build issue?
• Set STUB = %$STUB% should expand to TWiki::Contrib::CliRunnerContrib

Thanks for maintainig this extensions!

-- PeterThoeny - 16 Jul 2008