Tags:
extract_idea1Add my vote for this tag plugin_api1Add my vote for this tag create new tag
, view all tags
I am working on the definition of a base Plugin.pm class for plugins, please add your suggestions below

[ I have moved this topic to the Codev web to widen the discussion -- AndreaSterbini]

Some Ideas on how to enhance the Plugins API

I am considering to implement some simple enhancements for the Plugin API:

  • smoothly move to object-orientation:
    • if the init function of a plugin returns an object instead than 1 then all following handler calls are made as method calls
  • support for delayed loading of needed plugin/libraries.
    • add to the Plugin documentation page a declaration of the syntax recognized. The plugin is loaded/initialized ONLY if the declared syntax is present. E.g.
      • Set SYNTAX = %CALENDAR%
      • Set SYNTAX = %CALENDAR{.*}%

... when I find time I should study if/how the AUTOLOAD mechanism could be used for the delayed loading ...

-- AndreaSterbini - 25 Mar 2002

One way to defer plugin loading is to require that all plugin syntax be of the form %PluginName:Function%. This has a nice resonance with some possible future xmlized version of TWiki

-- DavidLeBlanc - 27 Mar 2002

It also avoids the otherwise inevitable and unresolvable problem of clashes in the plugin invocation namespaces. So, I support David's notion.

-- MartinCleaver - 15 Aug 2002

These ideas sound great. I'm getting concerned at the degrading performance of our TWiki, and believe a large part of the slowdown is due to the increasing number of plugins we are using.

-- MartinWatt - 07 Sep 200

Object-orientation: Could be done. I would put special attention to performance, could be to expensive.

Declare syntax: Sounds like a good idea to improve the performance. Plugins without the SYNTAX declaration should be loaded always (for compatibility). Some Plugins already bail out early; not sure how much performance gain we get compared to existing Plugins that return 0 on initPlugin.

Name clash: I do not see a problem as long as the Plugins introduce sensible names, e.g. the CalendarPlugin has %CALENDAR{...}%, and should probably not introduce a %CHART{...}% command.

Performance problem: Plugins can be written in a way that you see only a performance impact in the percent range. Before I enable a Plugin at work I always measure the performance without / with the new Plugin on a simple topic without Plugin commands, a complex topic without Plugin commands, and a topic without Plugin commands. For example I made performance improvements to the SmiliesPlugin before I installed it at work. For testing you can also disable all Plugins in TWiki.cfg to measure the performance withou any Plugins.

BTW, probably better to move this topic to the Codev web, the Plugins web is here for Plugin development, Plugin API enhancement should be discussed in the Codev web.

-- PeterThoeny - 08 Sep 2002

Nameclash: so, given that there is no way of programmatically asking plugins what names they use and therefore no way of an admin determining whether they have clashes in their system, it's okay to leave it to chance?

Leaving it to plugin authors to select names that don't clash might be fine if there was only one author writing pluings. Else they need to know what everyone else is using. Many words have multiple meanings, so that's one source of clashes. In other cases there are plugins that overlap in functionality so of course they are going to want the same names, that's another source. The namespace for directives should work the same way as perl's/CPAN variable Exporter package works.

Performance: I would guess most people don't do those checks, not least because they don't have your expertise. Can you provide a how-to? Better still would be a test harness to automatically test them when they get uploaded to the plugins web.

Moving this topic to Codev: I agree it should be. I tried to move it but I don't have the rights. I think MikeMannix might have moved all plugin related conversations to this web earlier, there are certainly others around here.

-- MartinCleaver - 16 Sep 2002

Maybe for start we can create a page like NameSpace, where authors can mention which names are taken, so others can select different names. It will not be perfect or automatic, but might be one central place to see if name is used by somebody else.

-- PeterMasiar - 17 Sep 2002

Nameclash: since (most) of the names are to be used by humans, their number and complexity have to kept as small as possible. If not even the developers get it straight -- how could a mortal writers do? So please -- no extra namespace prefix or such! Peter Masiar's idea of NameSpace topic for developer coordination sounds good.

Performance: If I understand correctly, every variable within TWiki and its plugins means, that each and every line is matched against each and every RegularExpression defining the variable's function. So processing time increases linearly with the number of variables/features defined - right?
How a about pre-parsing each line and then parsing only the applicable variables for full? A very primitive approach could be just the first chararcter of the variable. I.e. of passing

foo %BAR% foo %FOO%
into hundreds of RE's, how about passing
%BAR% foo %FOO%
into the "%B-handlers" and
%FOO%
into the "%F-handlers"? The number of REs to check would decrease by a factor 26, assuming even distribution of characters. Maybe 10 with a realistic distribution. Additionally, each RE could be anchored at the begin of the pre-parsed line, with considerable speedup. Is there anything in TWiki parsing, which rules out such an approach? Look-behind or such?

Wondering -- PeterKlausner - 23 Sep 2002

With reguard to performance, one or two RE's could do all the detection and extraction of variable names, so that a more intelligent function could figure out what to do next. By requiring all plugins to register the variables which they affect, and the names of the functions which process them, the plugin handler would merely have to extract variables from the text using a generic RE, and call the subroutines which those variables have been mapped to. If the variable requires arguments, they could be passed to the handler subroutines as a hash. Not only would this be an elegant solution, it would perform much better than the current implementation.

Example (somewhere in the bowels of TWiki::Plugins):
$text=$_;
$oldtext="";
while($oldtext ne $text) {
  $oldtext=$text;
  if($text =~ /\%([A-Z_:]+)({[^}]})*\%/) {
    $var=$1;
    $args=$2;
    $sub=&TWiki::Plugins::getPluginHandler($var);
    %argshash=&TWiki::getArgs($args);
    $tmp=&$sub(%argshash);
    $text=~s/\%${var}${args}\%/$tmp/;
  }
}

Of course, this example disreguards the order in which some variables should be processed.

As far as nameclashes go, namespaces seem like a good idea, although painful to type in text.

-- PeterNixon - 02 Nov 2002

I suppose precedence could be handled by creating precedence groups, e.g. what variables should be handled in which order. The INSTALLEDPLUGINS variable could be used for this (I think it is already, sort of):

Somewhere down in TWiki::Plugins, the order could be taken into consideration thusly:

@pluginorder=TWiki::Plugins::getPluginOrder();
foreach $plugin (@pluginorder) {
  $text=$_;
  $oldtext="";
  while($oldtext ne $text) {
    $oldtext=$text;
    if($text =~ /\%([A-Z_:]+)({[^}]})*\%/) {
      $var=$1;
      $args=$2;
      if($plugin eq &TWiki::Plugins::getPluginName($var)) {
        $sub=&TWiki::Plugins::getPluginHandler($var);
        if(ref($sub) eq "CODE") {
          %argshash=&TWiki::getArgs($args);
          $tmp=&$sub(%argshash);
          $text =~ s/\%${var}${args}\%/$tmp/;
        }
      }
    }
  }
}
# and then another loop to clean up everthing else

$oldtext="";
while($oldtext ne $text) {
  $oldtext=$text;
  if($text =~ /\%([A-Z_:]+)({[^}]})*\%/) {
    $var=$1;
    $args=$2;
    $sub=&TWiki::Plugins::getPluginHandler($var);
    if(ref($sub) eq "CODE") {
      %argshash=&TWiki::getArgs($args);
      $tmp=&$sub(%argshash);
      $text =~ s/\%${var}${args}\%/$tmp/;
    }
  }
}
$_=$text;

-- PeterNixon - 03 Nov 2002

Implementing precedence goes some way to solve the problem but I think it is not an end in itself because:

  1. it permits members of the TWiki community to go in separate directions when people implement different precedent orders.
  2. A plugin written now can clash with a core feature added later.

IMO, a better way is to mimic what Perl's Exporter package. (See below)

The alternatives require TWiki to search the core and all the plugins to get something to process a directive. This would become too slow as the number of plugins installed on a single site exceeds a threshold.

I propose invoking a plugin should be explicit, with a format such as:

  • X:Y:...

Where:

  • 'X' is something short that separates the namespace of plugin WikiML from the namespace of core WikiML. I'd propose %PLUGIN:...% but it is too long.
  • 'Y' is the name of the plugin (with (or without) the word PLUGIN stripped from the name)
  • 'Z' is optional and if stated is the name of the exported routine in the plugin that does the thing you want to do. If not stated it runs a default routine.
  • '...' are the parameters to that routine

Example:

(CALENDERPLUGIN) CALENDER{month="2" year="2002" bgcolor="cyan"}% becomes:

  • X:CALENDER:{month="2" year="2002" bgcolor="cyan"}% which is short for
  • X:CALENDERPLUGIN:{month="2" year="2002" bgcolor="cyan"}% which is in turn short for
  • X:CALENDERPLUGIN:default{month="2" year="2002" bgcolor="cyan"}%

For backwards compatibility or for those who like such things, we could also have a lookup table to wire the old format to the new. This is akin to Exporter's IMPORT directive.

Maybe on InstalledPlugins:

directive shortname webs active in Plugin invocation
% CALENDER{(.*)}% .* X:CALENDERPLUGIN:default{$1}%
% CHART{(.*)}% .* X:CHARTPLUGIN:CHART{$1}%
% BUG{(.*)}% Bugs, Main X:BUZILLA:BUG{$1}
% BUGLIST{(.*)}% Bugs X:BUZILLA:BUGLIST{$1}

Thus clashes are dealt with explicitly but can't happen accidently.

For those who think they don't care about clashes, this syntax could be added:

directive shortname webs active in Plugin invocation
% (.*){(.*)}% .* X:$1:{$2}

For the latter to work though there must be a way of introspecting the plugins for the names of the directives that plugin implements. This would work in the same way as the EXPORT_OK variable in CPAN's Exporter.

With regards to the performance issue:

  • PeterT: MartinWatt says he is having a problem. Can you at least see that it is theoretically possible, with say 100 plugins active?


See the CPAN:Exporter . [just follow the link AndreaSterbini] %PERLDOC{"Exporter"}%

-- MartinCleaver - 21 Dec 2002

I want to write a plugin in the C language. How can I do that?

-- WolfgangSpraul - 04 Apr 2003

PeterNixon: can you elaborate on which EnhancementsToThePluginAPI you are waiting for? We should target people's efforts and get MegaTWiki merged in before Cairo further progresses and the costs of merging escalate.

NB. Wasn't this subject going to be moved to Codev?

-- MartinCleaver - 25 Jul 2003

Wolfgang: Why would one want to write a plugin in C? Efficiency? If it's so, I guess you can always make a Perl extension, and write a little Perl Plugin that imports the C extension, acting as a wrapper.

-- EstebanManchado - 30 Jul 2003

I'M BACK!!! smile

I've just extended Plugins.pm to call Plugins written as objects (and instantiated by initPlugin).

I am reading your suggestions above on the pre-declaration of tags ... tomorrow I will try to implement a very very simple declaration scheme to gain some efficiency.

Note With delayed loading of required packages (e.g. see CalendarPlugin and PrologPlugin) usually a plugin hits only pages containing its tags.

Here is my current version:

-- AndreaSterbini - 18 Aug 2003

PrologPlugin shows a first example of tag declaration used (for the moment) only to check the tag syntax and to produce a nice table.

You need the Plugins.pm attached there

TODO:

  • move useful parts from PrologPlugin to Plugins.pm
  • use declared tags to speed-up plugins (it is not clear to me how to do what is suggested above and keep the current (lack of smile ) flexibility ...

-- AndreaSterbini - 21 Aug 2003

Good to see you back Andreas. I also noticed that Peter deleted your comment on CoreTeamHallOfFame in which you said you didn't have much time to give.

Have you had a change of work or something that affords you the time to make this a permanent stay? It would be great to have you around as much as you used to be.

-- MartinCleaver - 22 Aug 2003

In some way, yes ... this year I am collaborating with FrancoBagnoli on a paid joint project using TWiki.

This, unfortunately, does'nt mean that I can take a permanent position because I will be rather busy teaching courses from next October to December. Then I will be back again.

I could give a hand on the Plugins stuff (I am experimenting with OO plugins) and/or continue doing TWiki Research .

I am almost ready with a working first implementation of:

  • a ProgramsPlugin (a generalization of PrologPlugin for running make, bash, gprolog ... you just edit the plugin page and decide)
  • a Plugin that:
    • saves the TWiki authorization settings (ALLOW/DENY CHANGE/RENAME/VIEW WEB/TOPIC) of pages and webs and the groups definitions in a Mysql database to speed up authentication (see next item)
  • an Apache authorization handler
    • uses the stored definitions for authentication at Apache level, this way
      • the authorization is handled by Apache without reading a lot of files
      • the authorization protects also the attachments
      • you can use WebDAV to edit attachments (it works like a breeze!)
  • an Apache handler using Apache::SessionManager to handle sessions both with cookies and with URI rewriting
  • a ModPerlize.pl script that puts all global variables in a hash and threads the hash through all rutines

-- AndreaSterbini - 25 Aug 2003

Andrea, as part of your OO plugins experimentation, did you develop a plugins base class that all OO plugins should inherit from?? Im curious in that I am thinking about a fair number of items if built into a base class would make developing plugins easier from a development standpoint.

-- JohnCavanaugh - 27 Aug 2003

Not yet ... I first wanted to be sure that the handlers code works (it does) and I have just written a plugin to test the idea.

Now I would like to enhance the API ... I should find the time to analyze the current plugins available around to gather some good suggestions.

In the meantime I am trying to standardize the definition of tags and parameters (and their values), both to generate documentation and to do checks on entered values. I am sure that some part of ProgramsPlugin could be a good base for the above mentioned Plugin.pm

Let start a design phase: everybody is invited to suggest methods for the Plugin.pm base class

+++ Suggested definition of the Plugin.pm class

method parameters proposed by votes why (docs)
readDefinition self AndreaSterbini 1 gets the definition of the tag parameters, defaults and allowed values from the plugin topic, this allows tag/parameters internationalization
parseTagArs self, args AndreaSterbini 1 parse the args (the one you find between curly braces) and sets values.
"arg" self, tag, arg, [value] AndreaSterbini 1 gets/sets a tag argument ... defined (and cached) through AUTOLOAD
getCachedValueOrComputeIt self, cachefile, subToCompute, dependencies AndreaSterbini 1 gets a value from a file (with Storable) if it's newer of all the dependencies, else compute and save it in the cachefile (see ProgramsPlugin implementation)

TODO: (just a reminder for myself)

  • use PluginsWithNonFuncCode as a suggestion for new handlers/methods
  • use aggressively the caching of the AUTOLOAD generated methods (in OO Plugins.pm and Plugin.pm)
  • write the doit script to define form actions as Plugins ... it could introduce a new handler.

-- AndreaSterbini - 28 Aug 2003

Written doit (it's in ProgramsPlugin) ... you can use it for ANY tag (and not only for the ones defined in the plugin).

- AndreaSterbini - 8 Sep 2003

... mmmm ... to know what tags the plugins can handle I must load all the plugins ... this wastes time (in non-mod_perl sites).

How can I speed up plugin init ... mmmmm .... I could

  • move all code beyond =__DATA__ =
  • use CPAN:SelfLoader
  • keep on this side just a variable defining the tag correspondence
  • or else: (better)
    • use the topic to declare the plugins tags with a line like the following one:
         * Set TAGS = ...
    • and then init the plugin only if the tags are contained in the topic

Then if we use some form of caching we avoid parsing all these damned topics ... (e.g. as I am doing in ProgramsPlugin)

Even better if we introduce caching in several points of the initialization:

  • just after the Preference parsing ...
  • after the User/Group parsing ...
  • after the plugins initialization

Or we could cache data across several html calls (with CPAN:Cache::Cache or similar)

Else ... use mod_perl (then we must get rid of global variables) (some caching would be useful anyway).

I'll do some experiments.

-- AndreaSterbini - 11 Sep 2003

I don't understand the statement "use mod_perl (then we must get rid of global variables)". TWiki works fine with mod_perl now - we've used it at work for over 2 years. Is this some extra way of using mod_perl that avoids multiple processes that you know about?

-- JohnTalintyre - 12 Sep 2003

With mod_perl (and more precisely with CPAN:Apache::Registry that gives the best performances) and global variables you can be unluky and have two different view running at the same time and sharing their global variables. While this can be useful to share init data or DBI connections, this is normally a problem in TWiki. We (at Rome and Florence Univ.) experience sometimes such problem, as we have a lot of students clicking at the same time to visit course's pages.

If you run, insted, mod_perl with CPAN:Apache::PerlRun (less performing than Registry) you have your modules loaded once and recompiled each time they are run, thus they not share global variables.

Moreover, if we get rid of global variables we will be able to run several twikis at the same time even with different configurations.

-- AndreaSterbini - 12 Sep 2003

I am looking at the Plugins init code, it goes through reading each plugin page to collect all the plugin preferences. I am implementing a caching mechanism of all the preferences.

  • I cache separately the values read from each topic
    • they are stored as a topic's attachement with name cachedPrefs.dat
  • read the cached values if they are available and they are newer than the topic
  • else read and parse the topic and store the values in cachedPrefs.dat
  • finally merge the values to respect the TWiki/Web/User preference order

The code is almost ready, I am testing it.

-- AndreaSterbini - 12 Sep 2003

I would like to see an interface to allow plugins to read and save topics. That is, EditTablePlugin uses private functions to do it's work. These should be public.

-- JonathanCline - 12 Sep 2003

Jonathan: this is already possible. Use readTopicText( ) and saveTopicText( ).

-- PeterThoeny - 16 Oct 2003

See my comments on EmptyOoPluginDev#efficiency

-- ColasNahaboo - 16 Oct 2003

Check PluginOOApi for some ideas and discoveries

-- RafaelAlvarez - 08 Sep 2004


It would be nice to be able to select which parts of the template a plugin applies to. I see by the StepByStepRenderingOrder that rendering plugins will get called at least 3 times (%text%, %metadata%, %template%). Is there anyway way to flag steps as "clean"? e.g. No need to call me here, I don't do metadata or templates, only %TEXT%?

Beside the obvious efficiency gains (not processing areas which don't get changed) this is also safer for the budding plugin devloper -- most of the trickier rendering bits revolve around skipping html tags, which are more likely to occur in the metadata and template areas.

-- MattWilkie - 23 Feb 2004


CategoryPluginsAPI
Topic attachments
I Attachment History Action Size Date Who Comment
Perl source code filepm Plugins.pm r1 manage 12.5 K 2003-08-18 - 21:45 AndreaSterbini Version calling also OO-Plugins
Edit | Attach | Watch | Print version | History: r33 < r32 < r31 < r30 < r29 | Backlinks | Raw View | Raw edit | More topic actions
Topic revision: r33 - 2004-09-08 - RafaelAlvarez
 
  • Learn about TWiki  
  • Download TWiki
This site is powered by the TWiki collaboration platform Powered by Perl Hosted by OICcam.com Ideas, requests, problems regarding TWiki? Send feedback. Ask community in the support forum.
Copyright © 1999-2017 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.