create new tag
, view all tags

Some TWikiVariables Should Be In Plugins

There are some TWikiVariables that have "complex" behaviors (%SEARCH%, %INCLUDE%, %TOC%). Now that in DakarRelease there is a mechanism in place to process plugins tags in n efficient way, I think it could be useful to move the handling of these complex tags out the core and into plugins.


I can think of three advantages:

  1. Reduce the amount of code in the core (less compilation time, simpler to maintain)
  2. The tag handling can evolve independently of the TWikiRelease, that is, users can benefits from new features added to existing tags (like the "sectional include" to the %INCLUDE% tag) without upgrading inmediately.
  3. There could be "pluggable" implementations for these tags. For example, the current %SEARCH% implementation depends on the topics being stored in a filesystem to use fgrep to perform the search. An alternative implementation of %SEARCH% can use Plucene to search the database-stored content.

These are the TWikiVariables that I propose we should move to plugins:

  1. %SEARCH%
  2. %INCLUDE% and all related tags:
    1. %BASETOPIC%
    2. %BASEWEB%
    7. %SECTION%
  4. %TOC% (this was suggested somewhere else)
  6. Date related tags:
    1. %DATE%
    3. %GMTIME%
  7. %IF%

-- RafaelAlvarez - 22 Jan 2006

These features seems so basic to me that 99% of all TWiki installations would use such plugins.

Dakar is much slower than Cairo already unless you are lucky and can use mod-perl. Will this initiative speed up the code or slow it down?

I bet it will slow it down. For next major TWiki release you have to consider speed and scalability also.

-- KennethLavrsen - 23 Jan 2006

maybe not, if only 2 of the tags are used on a topic, only 2 of the handlers should need to be compiled - Dakar has a couple of plugins that seperte out the registration code from the main - this would be the same for these core plugins. its not so much to remove them from the core distribution, its more to move them out of the core code, so they only conditionally slow twiki down..

it also makes TML rendering and processing more consistent if all TML including optional plugin TML is handled in the same way

-- SvenDowideit - 23 Jan 2006

I bet that performance will, at worst, stay the same.

The current mechanism to handle these tags is to register a handler that calls code in another module. The mechanism that should be used if they're move to plugins is to register the tags in the "init" method. If the plugin are coded the right way, the code that will actually handle the processing of th tag could be made to be compiled "on demand".

-- RafaelAlvarez - 23 Jan 2006

Also, from the Architectural point of view, this is a step to move TWiki to be implemented around a microkernel with several pluggable features. If done the right way, it can be made as fast as needed while being very flexible.

-- RafaelAlvarez - 23 Jan 2006

Here here. I agree. If TOPICLIST had been a plugin, then I wouldn't have had to make a plugin to add a feature to it. Of course, I could have done it in TWiki.pm, but it was a bit overwhelming and the timing was poor.

Let's do this for 4.1.

-- MeredithLesly - 23 Mar 2006

MeredithLesly started to create some Plugins in the TWiki 4 branch with the goal to externalize Plugins. This needs to be discussed and agreed on before work is done in TWiki 4. All major work should be done in DVELOP, after we agree on the scope of work.

Like with everything, there are advantages and disadvantages with externalizing core variables into plugins. There are some major disadvantages I see, based on the TWikiMission:

  1. Not KISS: Maintenance overhead, testing overhead, documentation overhead
  2. Compatibility: Evaluation order changes, which can break TWikiApplications. For example, TOC is done last, after all internal and Plugin variables (and recursively internal again) so that it can act on headings created dynamically
  3. Compatibility: It is more likely that Plugin authors change syntax
  4. Compatibility: Existing applications assuming the current set of variables will break if a Plugin is disabled or not installed
  5. Maintainability: Plugins should be able to run on a number of TWiki versions, which cannot be done with variables that depend on a particular TWiki version
  6. Performance: Plugin invocation (compile, init) is slower compared to core code

The Plugin framework already offers the flexibility of overloading internal variables, e.g. if someone does not like the spec of a variable it can be redefined.

Crawford already did a lot of work to clean up and modularize the variable handling, so maintaining them in the core is not that difficult. Not sure if the variable handling can/should be further modularized in the TWiki core. One possible thing is to selectively load rarely used variables that are large (to address the advantage RafaelAlvarez pointed out.)

-- PeterThoeny - 31 Mar 2006

The title is wrong. It should be ALL not some.

Simplify, sinplify, simplify. Make all TWikiVariables, whether they are core or resut from plugins be handled (and 'registered') the same way.

Peter, I disgaree that the disadvantages you present are in the elast bit valid. If we just move them, yes, but with a suitable architectural design they become advantages.

Modularization with the correct architecure IS KISS. For example, having a directory /lib/twiki/variables/ in which the load-on-demand version of what are presently the core variables. If the parser encounters %TAG% and registration table doesn't have an entry for ='TAG' then /lib/TWiki/variables/ is searched for TAG.pm.

This form of modularization leads to better isolation of functionality and hence clearer functional decomposition and hence ease of documentation and maintenance.

Any compatability problems that arrise will b from an incorrect implementation or lack of consideration when building the registration table.

Performance? I think it will improve becuase it is a compile on demand and we have determined that compile time is the limiting issue with TWiki. Yes, if you disk is slow then loading these modules, any modules, any modules at all!, will result in poor perfromance. But that's a hardware issue.

The SIMPLIFY is to treat all variables & tags the same wherever they come from. This gets back to Crawford's table for registering them.

-- AntonAylward - 31 Mar 2006

Sanitized modularation needs a well designed dependency tracker. Any module has to define (and register), which other modules it needs. This way, you can disable all modules that have not resolved dependencies or vice verse auto-enable all modules thar are neccessary.

-- TobiasRoeser - 31 Mar 2006

Peter, I don't see your points as disadvantages, I prefer to call them "risks" as they are just hipothesys not yet supported by facts.

Maintain the plugins for variables is not a bigger overhead than maintaining all the plugins that are needed for the TWiki web to work. Most of the time, they won't change as they shouldn't be calling any function outside the Public API in Func.pm (and yes, we should extend the Func.pm module to support all and every tag as plugins)

Compatibility is a valid point, and is a big risk. This can be mitigated is these plugins are treated diferently at the svn repository level: Only those with full checkin rights may modify them. So, only those that can modify the core should be able to modify the plugins. I think that we are responsible enough not to cause a big "mayhem" :). Also, these plugins should be shipped installed and enabled by default (much like TablePlugin, EditTablePlugin, etc).

As for performance, with CrawfordCurrie changes to the parser so plugins can register their tags, processing a plugin tag is as costly as calling a core tag. Actually, some of the variables are actually being handler outside TWiki.pm (SEARCH and IF come to mind).

-- RafaelAlvarez - 31 Mar 2006

Actually, I see what MeredithLesly has done is quite good. It allows TWiki4 users to use tags that perhaps only will be present after Edin*. Is much like the IF construct that NatSkin has, it can be seeing as a duplicate from the IF in the core, with the added advantage that they can be used in OLDER versions.

And THAT is the key advantage, variable syntax CAN evolve independently of the core, and CAN be build in a way that are backward compatible so the can be used in older versions (perhaps).

By letting those var be manage by plugins, we encourage other plugin authors to provide different implementations for some (METASEARCH, SEARCH, IF, INCLUDE come to mind). And the user will have the option to choose whatever implementation he/she think is the best.

-- RafaelAlvarez - 31 Mar 2006

I finished reading the whole stuff in the mailing list and there is a point that was somehow obfuscate in the discussion: Having to maintain about 40 plugins in the installation. And I think the answer to that concern is somewhere above in this topic: "Core" plugins should be "discovered" at runtime. Perhaps. Or we can find some mechanism for that.

This whole idea of having a microkernel with thousans of "plugins" is not new, and is proven to work, proven to be quite efficient if the plugin implementation is done right, and provent to be a good architecture (if done right). As I said before, time to stop preaching and start working. Those of us that think is a feasible way to go, must prove it by code. Those of you that think is not feasible, wait until the code is there and prove that is not feasible.

I'm really, really tired of never-ending-long discussion about a thing, throwing arguments at each other, and never actually DO something.

So, MeredithLesly threw some code to the repo. Is someone out there out to the task to measure the performance impact of having those plugins active vs having then not active in the case that the topic contains and don't contains the tags?

-- RafaelAlvarez - 31 Mar 2006

There is another key-point to keep in mind. What about documentation? Most of current default plugin variables e.g. EDITTABLE can not found in TWikiVariables. This problem will even grow, if more and more variables move to plugins.

I like the idea of modularization, but from the beginning of development, we/you have to keep in mind, that documentation has to be plugable too. This means it has to be automatically available if the plugin is in use, else not, without any impact (in complexity) to the user. Any plugin should export some kind of docs itself or at least the documentation should be dynamic compiled depending on (in-)activation of the corresponding module/plugin.

Just my 2 cents. smile

-- TobiasRoeser - 31 Mar 2006

If I can allocate 100 precious hours for TWiki coding, what brings more value to the TWiki project overall: Externalize internal variables vs. creating user centric innovations with perfomance in mind? See more at DevelopersVsUsersDiscussion.

Now, on the other side I do support experimenting with the code. If the 6 points I layed out above are addressed we can accept that into the core. May be we need multiple branches for experimentation to see how things work.

-- PeterThoeny - 31 Mar 2006

Tobias, plugins can be self-documenting via the PodPlugin (or whatever it's called).

-- MeredithLesly - 31 Mar 2006

Peter, if I can allocate a few precious hours for TWiki coding, what brings more value to the TWiki project overall: trying to find ways to speed things up or adding new features, user-centric or not? Right now, I think many people would vote for the former.

  • See my point: 6. Performance: Plugin invocation (compile, init) is slower compared to core code (I am happy to be proven wrong) -- PTh

-- MeredithLesly - 31 Mar 2006

Created a branch in SVN for experiments stemming from this topic:

http://svn.twiki.org/svn/twiki/scratch/SomeTWikiVariablesShouldBeInPlugins/ (Branched from TWiki4, r9623)

-- SteffenPoulsen - 31 Mar 2006

Thanks Steffen for creating the experimental branch!

-- PeterThoeny - 31 Mar 2006

According to Alan Cooper, developers will continue to fix things which are not broken until they are broken. Nice development work is going on here.

I'm really glad that this stuff of revision 9624 has been moved to a separate branch, and I hope it will be removed from 4.0. Even if is as "compatible" as it can get it is going to slow down both installations without mod_perl (because both Twiki.pm and the plugin have to be compiled) and TWikis with mod_perl (because every request calls an extra bunch of initPlugin). There are a couple of other problems with the approach, and some have been written down (read, for example, PluggableDocumentationArchitecture). Quoting Kenneth: I am short of words.

-- HaraldJoerg - 31 Mar 2006

One thing that people seem not to have considered is that starting with Dakar there's very little difference between a TWikiVariable-like thing and a plugin that just registers a tag. Each has a tag associated with a function which is called when the tag is parsed. A full-blown plugin, on the other hand, needs to get in via one or more handlers which are called at (theoretically) known points in the TWiki code flow and is a very different animal.

So perhaps instead of freaking out over the idea of moving TWikiVariables into plugins, perhaps extensible/overridable TWikiVariables that would be configured differently than plugins should be considered. Off the top of my head, one way to do this is to do it at configure time, with a file containing the necessary information written out to be read by TWiki.pm and/or whatever else needs to read it or, alternatively, all of the code for the selected TWikiVariables assembled into one file, depending on performance evaluation. This would allow the person configuring the TWiki installation to choose among different implementations of the same tag, as well avoiding (some/many/most) name conflicts.

-- MeredithLesly - 01 Apr 2006

Folks, I'm sorry I haven't been keeping up, or communicating clearly, but this is not a good idea - yet. Peter already made some of these points, but I'll re-iterate.

Firstly, Moving TWiki variables into plugins is bad news for performance. Each plugin added degrades runtime by 3-5% irrespective of whether it does anything or not. If variables are to be housed in extensions, they must be in contribs, not plugins.

Secondly, I don't think we have the infrastructure to support this change. If we move all these functions into extensions, we end up with a maintenance nightmare with the current processes. Basically, the current implementation of the core TWiki variables use APIs within the core that are not published. If we create a slew of extensions that leverage these APIs, we will rapidly find that changing the APIs is impossible without changing every extension. If we have to change every extension, why should they be extensions? We don't have the processes in place to revision extensions in lockstep with core APIs, and the APIs themselves are definitely not mature enough to be frozen (even Func isn't mature enough to be frozen).

Finally, it's human nature to want to extend TWiki, so I lose sleep over the idea that some script kiddy could redefine a core TWiki variable and then ask for support when their site goes pear-shaped. We don't have the infrastructure in place to know what has/hasn't changed in an installation (Martin did some great groundwork on this, but no-one has followed up). We also don;t have the infrastructure to support simple load/unload of these modules.

I don't think we need to go as far as Sven has suggested, and have variables defined by grammars. However I do think we need some minimal infrastructure:

  1. The ability to load extensions dynamically from the core (see below)
  2. The ability for contribs to dynamically include documentation
  3. Completion and full documentation of the TOM core objects
  4. Infrastructure to know what has changed in a release
  5. Simple infrastructure to support loading/unloading of efficient contribs (c.f firefox extensions)

So, in summary, this is a great idea, and indeed is the original idea behind modularising the variables. But we can't rush at it; there is a lot of groundwork required to be finished first. Frankly, I really, really, really don't want to be the one left to clear up the mess if this is done prematurely.

How to load extensions dynamically from the core

For example,

in LocalSite.cfg:

$TWiki::cfg{Variables}{MAKETEXT} = 'TWiki::Contrib::MakeTextContrib';
In TWiki.pm, in the BEGIN block (so that it is precompiled under mod_perl):
foreach my $var ( keys %$TWiki::cfg{Variables} ) {
    eval "use $var";
In (e.g.) MakeTextContrib.pm:
    $TWiki::constantTags{MAKETEXT} = \&_maketextVariableHandler;

-- CrawfordCurrie - 01 Apr 2006

I should point out that I didn't move the TWiki variables, nor did I do it to all of them. All I did do was copy the code of a few that didn't use unpublished API and put that code into plugins. The only difference in the code was to change "this" to "session". It's a way to see what impact on performance using (some) TWikiVariables as plugins would have.

CC: Each plugin added degrades runtime by 3-5% irrespective of whether it does anything or not.

Isn't that a serious problem in and of itself? That means that if a TWiki uses a plugin in a single topic every single page will be slowed down by 3-5%. If that, in fact, is accurate for all plugins. My timing tests showed that at most a simple page display was somewhere between 2% and 4% (somwhere between 30 to 50 ms) slower with the 6 plugins enabled (and the code still in TWiki.pm, of course). Obviously we want things speeded up, not slowed down, but my experiment showed that using plugins rather than TWiki.pm code for at least some of the variables would at worst have minimal impact on speed.

As to why I did it against (what I thought was final) 4.0.2, the reason was that I wanted to test against the current version, not against whatever changes people have made in DEVELOP, changes I am unfamiliar with as I've been using the 4.0.x svn exclusively.

-- MeredithLesly - 01 Apr 2006

There are plugins and there are ...

Well, you are correct, Crawford, plugins have an overhead. But lets be clear:

The current legacy architectural decisions impose a structure whereby plugins HAVE to have a significant overhead.

Once upon a time, operating systems considered processes to be "heavyweight" items. There might be only one process (MSDOS, batch on mainframe) with a 'transient program area' into which programs were loaded one at a time, or a small and fixed number of processes would be created at sysgen time (VMS, various IBM mainframe OSes).

The along came UNIX with lightweight processes. The shell would spawn processes with wild abandon. Spawned processes were intrinsic to the architecture. Think of the INET deamon (http://www.xinetd.org); it listens and spawns off processes to do the actual work. Additional facilities can be "plugged in" to this lightweight system. By contrast, the equivilent on something like the Tandem is one monolithic process with all the facilities built in. We see a similar setup with database tools like CICS - one monolthic program that hadnles everything, database, application, telecoms. Under UNIX this wold be dozens of cooperating processes, each doing one thing, doing it well, with well defined interfaces. (Does this sound like a precursor to OO? Well isn't the shell a OO language - methods, messages, polyorphism ...?)

During the 1980 when the VAX came out and Bill Joy was battling with David Cutler of DEC over which was more efficient, UNIX written in C or VMS written in assembler (Joy won. Cutler went on to say he'd never write an OS in assembler again); one simple 'array of bytes' file system or many file types (again UNIX won) ... one of the measures of a good UNIX implementation was how fast it could spawn processes.

But eventually it was decided that even processes were too heavyweight and along came threads.

Lightweight not-a-plugin plugins are load-on-demand

Crawford's code fragment above that inserts the handler into the constantTags has is a good example of 'lightweight'. However his eval loop proves his point that this is more expensive. This is what I mean when I say:

The current legacy architectural decisions impose a structure whereby plugins HAVE to have a significant overhead.

Now suppose we think for the moment of the large set of variables that simply substitute text:

  • Date related tags
    1. %DATE%
    3. %GMTIME%
  • HTTP & header stuff
  • Current and possible languages
  • Names of webs and key pages
  • Environment variables
  • User name and info
  • ... and many more

These are quite different from variables-that-are-functions such as

  • %SEARCH%
  • %IF%
and from the more complex operations such as

Please note that documentation on these is already modularized.

My point here is that load on demand tags are different in nature and complexity to plugins and tags that need to be registered.

The codebase already differentiates between the built-in tags and the plugins. The plugins require:

  • (possibly) complex init at the mod_perl level
  • (possibly) complex init per topic operation
  • registration of the plugin and possiblt the tags it uses
  • handler code for each tag, plus possibly
  • comon tags handler, as well as before and after handlers
  • pre and post rendering
  • before and after edit
  • before and after save
  • before and after attachemetn save
  • merge resolution
  • modify HTTP header
  • redirect query
  • form fields

Reality is that few of these are needed by most plugins. Most plugins could be lightweight. Some, for example UserInfoPlugin could be implemented in parts with a shared library using the load-on-demand model I discussed earlier, since they are doing straight forward text substitution.

The bottom line:

With a suitable architecure unused and unwanted parts of the TWiki core can be made into load-on-demand and hence their compilation avoided or deferred.
Load-on-demand is different from the current plugin architecture.

-- AntonAylward - 01 Apr 2006

Extreme Example of L-o-D

Let me give an extreme example of load-on-demand. Note: I'm not proposing this as an architecture, I'm just using it to illustrate the difference and to highlight the point that long ago Peter made design decisions which we've internalized, assumed and never questioned. Different design decisions could have been made. "This is is the story of one of them ..."

Imagine a TWiki that has no code in the core to handle any variables or tags. None at all. So you view a page. This page has no variables in in so all is fine. Pages with no variables are common. Some wikis don't even have variables! No overhead of loading plugins so its FAST!

Now we load a page with a variable in it - %SOMETHINGOROTHER%. The render code sees this and recognizes its a tag. It hasn't seen any tags so far so the tagHandling code goes into work.

The first thing the tagHandling code does is load the tagMappingTable which is a config file. (Lets leave aside how this is generated for the moment, just say its automated.) This table maps tags to the modules that handle them. The tagMapping finds the right entry and loads the module and off it goes. Load on demand. Only the plugins needed to handle that tag are loaded, so no others are loaded, hence its FAST!

Suppose the module actually needs other things loaded - it has dependencies. All well and good, we deal with that as suits the situation. Either the module has its own require statements in the code or the mapping table has a list of modules to load.

The point of this illustration is:

  1. There are other architectural decisions that could have been made
  2. We don't have to load all the code at the begining
  3. We don't have to load code we're not going to use
  4. We don't need to load the code we do use until we actually need to use it.
  5. Table driven methods are cool cool!

-- AntonAylward - 01 Apr 2006

Mant plugins are already lazy loaded. The method I gave above could easily be recoded to lazy-load, just by providing a facade around the tag handler.

$TWiki::cfg{Variables}{TOC} = TWiki::loadTagHandler('TWiki::Contrib::TOCContrib', 'TOC');

sub loadTagHandler {
    my( $module, $tagName) = @_;
    # create an anonymous lazy-loading function that will be replaced with the real
    # function the first time it is run.
    my $fn = $module.'::'.$tagName;
    return sub {
        eval "use $module";
        $TWiki::constantTags{$tagName} = \&$fn;
        return &$fn(@_);

But this is mere technicality, and isn't addressing my main point. The infrastructure to support dynamic variable definitions does not exist. This is far more important (though intrinsically less interesting) than the technical detail.

The existing Plugins interface (Func + handlers) may be a bunch of chains around TWiki's neck, but can't be cut free. We need to find a way to support more efficient ways to plug code in, but without compromising the current plugin mechanisms. My chosen approach is to rationalise and freeze key internal interfaces/objects to allow alternate implementations to be plugged in on demand. These alternate implementations will include tag handler implementations, though this just one interface of many (and IMHO one of the least important, as it is already fairly well understood). I'd far rather people spent their valuable brain-watts thinking about how to clean up the other internal interfaces that I had to bodge around (preferences, users, store) and how to further abstract out internal mechs like access control.

-- CrawfordCurrie - 02 Apr 2006

Edit | Attach | Watch | Print version | History: r25 < r24 < r23 < r22 < r21 | Backlinks | Raw View | Raw edit | More topic actions
Topic revision: r25 - 2006-10-08 - SvenDowideit
  • Learn about TWiki  
  • Download TWiki
This site is powered by the TWiki collaboration platform Powered by Perl Hosted by OICcam.com Ideas, requests, problems regarding TWiki? Send feedback. Ask community in the support forum.
Copyright © 1999-2018 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.