Implemented: Dependency checking function TWiki::Func::checkDependencies for Plugins

Plugins and Addons have an ongoing problem with dependencies on shared or common code that for one reason or another is not in the TWiki core. This topic discusses this problem and proposes a solution.

The problem arises when two plugins need to share code to perform and identical function. A classical example of this is the Attrs.pm attributes parsing class that is used by multiple plugins.

A first reaction might be that such common code should be moved into the core. However there are a number of problems with this approach:

1. It's very difficult to retrofit a plugin using the code to an earlier TWiki release
2. Inclusion of new code in the core is very slow; it can take months to get a proposal reviewed, and then years to get it included in a stable release - if ever.
3. The functionality may not be required outside the plugins; in this case, the code is simply baggage on the core for most users.

So, a strategy for shared code in plugins is required. Some requirements on this strategy:

1. It must support the concept of "unzip installation"
2. It must have a minimal negative effect on performance
3. It should not require recoding of any existing plugins that have no dependencies.

The first point rules out an "installation process" for checking dependencies, such as CPAN. So dependencies have to be checked at runtime and resolved manually by the installer, and this has to be as light as possible.

We can recover the version of installed plugins easily, because of the VERSION global variable every plugin carries. But we can also simply check for the presence of a module; this allows us to perform a consistent check for installed CPAN or other perl modules the plugin requires.

The proposal is to introduce a simple few lines into the initPlugin method of plugins with dependencies, as follows:

my @dependencies = (
  { package => 'TWiki::Plugins::ActionTrackerPlugin', constraint=>">1.005" },
  { package => 'TWiki::Attrs', constraint=>"==1.00" }
  { package => 'Time::ParseDate', constraint=>undef }
);

sub initPlugin {
  ...

  my $depsOK = 1;
  foreach my $dep ( @dependencies ) {
    my ( $ok, $ver ) = ( 0, 0 );
    eval "use $dep->{package}";
    unless ( $@ ) {
   if ( defined( $dep->{constraint} )) {
       eval "\$ver = \$$dep->{package}::VERSION;\$ok = (\$ver $dep->{constraint})";
   } else {
       $ok = 1;
   }
    }
    unless ( $ok ) {
        my $mess = "$dep->{package} ";
        $mess .= "version $dep->{constraint} " if ( $dep->{constraint} );
   $mess .= "is required for $pluginName version $VERSION. ";
        $mess .= "$dep->{package} $ver is currently installed. " if ( $ver );
        $mess .= "Please check the plugin installation documentation. ";
        TWiki::Func::writeWarning( $mess );
        print STDERR "$mess\n";
        $depsOK = 0;
    }
  }
  return 0 unless $depsOK;
  ...

The eval prevents failure if the plugin is not installed. If the initPlugin checking code were in Plugins.pm the init code could be reduced to:

   TWiki::Plugins::checkDependencies( \@dependencies );

This way the most useful constraints can be expressed and tested at plugin initialisation time. Errors are reported to warning.txt, and repeated to STDERR for output in the server error log.

This syntax (the dependencies in an array) is amenable to automatic extraction of dependencies by build tools such as Build.pm, which in turn can help build the documentation for the plugin.

Note that this proposal handles runtime dependencies only. Compile time dependencies, such as those on the common test fixtures, or on the build system, will not be resolved this way. This is a minor issue.

Note: this can also be used to express the standard plugins module version check:

my @dependencies =
  (
   { package => 'TWiki::Plugins', constraint => '>= 1.010' }
  );

The following actions are required to make this all a reality:

1. The above code needs to be incorporated in the EmptyPlugin and DefaultPlugin to act as an exemplar. or the check needs building into Plugins.pm
2. This topic needs to be refactored as documentation of the process.
3. A new class of topic needs to be born in the Plugins web viz "Plugins.CodeLibrary"
4. SharedCode needs to be reclassified as a CodeLibrary
5. The Build module and test fixtures need to be moved out of SharedCode into their own CodeLibrary
6. The Build module needs to be extended to automatically pick out these dependencies
7. Ditto the CairoCompatibilityModule

Contributors: CrawfordCurrie, PeterThoeny

This is now working for:

Plugin	Depends on
CommentPlugin	SharedCode
ActionTrackerPlugin	SharedCode, Time::ParseDate
DBCachePlugin	SharedCode
FormQueryPlugin	ActionTrackerPlugin, DBCachePlugin

versions in CVS.

At the same time I refactored the plugins so they all now do lazy-eval of any modules they depend on, improving startup times by a few fractions of a second.

Attached is a patch to FuncDotPm that adds checkDependencies. Note that this has to be done as a function call from the plugin, and not from any more "hands-off" mechanism, because dependencies should only be checked in the time and place of the plugin author's choosing - i.e. when they are actually needed, and not before. I use this for example to check dependencies in lazy initialisation functions that don't execute until the plugin's tags are encountered.

I'm really keen to have this in Cairo, and it is extremely low risk because it is simply the addition of a method to Func. Please, please, please put it in - otherwise I'll have to duplicate the code in every plugin!

-- CrawfordCurrie - 06 Aug 2004

This is now in SVN, thanks Crawford and Sven. I changed the POD doc example from ( TWiki::Plugins::VERSION >= 1.030 ) to ( $TWiki::Plugins::VERSION >= 1.025 ) and bumped the TWiki::Plugins::VERSION up by one to =1.025. The version number was off; the sample code with missing "$" was incorrect and produced a warning.

Question: Is there a perfomance impact if 3 different Plugins do a dependency check on the same module (e.g. Time:Parse) since the eval "use ..." is called three times?

I also fixed the white space and indentation. Please follow the guidelines, see ReadmeFirst

-- PeterThoeny - 07 Aug 2004

I did some testing and found an issue: Apache error logs contain also the error messages from the eval "use..." in case the module is not there, e.g.:

BEGIN failed--compilation aborted at (eval 31) line 2.

Crawford, can that be fixed?

All this testing and doc review takes time. Going forward, lets be more strict in accepting last minute changes. (Although we could release Cairo with the current code since it is a minor issue.)

-- PeterThoeny - 07 Aug 2004

This is designed behaviour. It is an error message, caused by an error, and deliberately printed to the error stream by the "print STDERR $@;" in the example code. The reason for echoing messages to the error log is that it is an error to have a non-functional plugin in the installation.

BTW, I provided a patch for some genuine spurious errors reported in error_log in PatchForErrorsInHttpdLog

In answer to your question above, perl is smart enough to know the module has previously been "use"d. The version number was off because I didn't know what version number you would be using in the release; sorry, I should have pointed this out.

-- CrawfordCurrie - 08 Aug 2004

Please see ContribPackage for the process for contributed code management.

-- CrawfordCurrie - 08 Aug 2004

OK, understood on error message. Implementation set to 100%.

-- PeterThoeny - 08 Aug 2004

Thanks Peter. The ContribPackage topic has been there for 5 days now without attracting any comment, so I propose to split up the SharedCode module into it's component parts, before too many downloads happen.

-- CrawfordCurrie - 13 Aug 2004