Tags:
caching1Add my vote for this tag performance1Add my vote for this tag create new tag
view all tags

VarCachePluginDev Discussion: Page for developer collaboration, enhancement requests, patches and improved versions on VarCachePlugin contributed by the TWikiCommunity.
• Please let us know what you think of this extension.
• For support, check the existing questions, or ask a new support question in the Support web!
• Please report bugs below

Feedback on VarCachePlugin

This Plugin is for you if you have a page that loads very slow because of TWikiVariables like %SEARCH{}% and %INCLUDE{}%. Enjoy.

An alternative is the FormQueryPlugin.

-- PeterThoeny - 16 Jul 2004

There is a typo in the VarCachePlugin page in that this is spelled %VARCACHR%.

I have difficulties understanding the documentation:

  • It is said that the == variable gets replaced with one of the above (in the documentation table) messages. What is shown if neither the cachemsg nor the updatemsg variable are present? I would expect that the variable is expanded into the default messages in that case.
    • If that is the correct interpretation, you might want to state that in the documentation,
      • To the default as indicated in the Default column -- PTh
    • however, this is not what I observe at VarCachePluginTest. From VarCachePluginTest it appears that no expansion takes place.
      • This Plugin is not installed on TWiki.org -- PTh
  • Also, what is unclear whether the cache is refreshed automatically after the indicated time, or whether the cache is refreshed on the next topic load after the cache expired.
    • On next topic load. As indicated, The cache gets updated automatically when looking at a topic that has an outdated cache; in which case the user will notice a slower page rendering then usual. -- PTh

-- ThomasWeigert - 16 Jul 2004

Thanks for the feedback, I will work on the docs. It can be hard to write good docs.

-- PeterThoeny - 16 Jul 2004

This looks like an excellent plugin - some of the other cache strategies presented, that caches the entire HTML etc, bring new headaches with skins etc - this idea sounds promising.

Two questions:

  • Can anything be done about the ModPerl issue (with $paramMsg) mentioned in VarCachePlugin.pm?
  • There is a security issue if this plugin is used in topics with protected information - topic text can now be retrieved through the /pub dir (i.e. .../pub/TWiki/VarCachePluginTest/_VarCachePlugin_cache.txt). I believe it's acceptable, but making people more aware of this effect up front wouldn't hurt, I think(?).

-- SteffenPoulsen - 01 Apr 2005

mod_perl issue: There is a potential race condition within variable expansion of a page (e.g. time taken between beforeCommonTagsHandler and afterCommonTagsHandler. If two pages are viewed and updated at the exact same time, one page might show an incorrect message. At this time I do not have a workaround. Note that this issue does not affect the actual caching, only the short cache result message.

Security issue: Yes, cache pages can be protected with a Apache config setting. Should be documented in the next release.

-- PeterThoeny - 03 Apr 2005

checked .zip into CVS

-- WillNorris - 19 Jul 2005

Does this plugin actually work in TWiki4?

I notice it is installed on this TWiki.org. Is the lack of TWiki4 compatibility a reason why twiki.org is still not upgraded to TWiki4?

-- KennethLavrsen - 02 Mar 2006

This Plugin plays by the rules and should work in Dakar, but I have not tried. (See PluginsConformanceReport).

On TWiki.org upgrade, mainly a question of allocating time to do it. Not sure of Sven solved the outgoing e-mail issue with the ISP. Also, we need to decide on the auth options, balance between ease of use and security.

-- PeterThoeny - 02 Mar 2006

I hav been using it with Dakar (both beta and final) for several months without problems.

-- LynnwoodBrown - 02 Mar 2006

Didn't work for me on Dakar (4.0.1) as it references the non-existant TWiki::getCgiQuery() function. Changed it to TWiki::Func::getCgiQuery(); and now it works like a charm smile see the attached VarCachePlugin.pm.diff

-- MarcLangheinrich - 06 Mar 2006

The ZIP file on the Plugin topic was out of date. I made a few more doc changes and updated the ZIP to the 06 Jan 2006 version.

-- PeterThoeny - 06 Mar 2006

http://koala.ilog.fr/twikiirc/bin/irclogger_log/twiki?date=2006-07-11,Tue&sel=959#l955

MartinCleaver: That  - its to make TWiki cache the search?
MartinCleaver: but it still registers as a hit?
PeterThoeny: yes
PeterThoeny: just the %VARS{}% are cached
MartinCleaver: is there a reason this not a meta data item?
MartinCleaver: (the directive to make it behave this way)
PeterThoeny: it uses the standard variable naming convention
MartinCleaver: I'd venture that a %META:VarCachePlugin:VARCACHE{168}% would be a better place for it.
MartinCleaver: it'd scare the living daylights out of any mortal editing the page

-- MartinCleaver - 11 Jul 2006

Suggested by MichaelDaum: Add rel="nofollow" to the refresh link to prevent robots from hitting the refresh link.

-- PeterThoeny - 20 Sep 2006

I used this plugin on an earlier version. We upgraded to 4.1 and it doesn't seem to work anymore which is quite unfortunate.

Can anyone confirm that it does not work on 4.1?

-- KaiMoorfeld - 11 May 2007

TWiki.org currenty runs 4.1.1 and this plugin works. Check your installation; please open a support question with details in the Support web if needed.

-- PeterThoeny - 11 May 2007

Nice to have feature/enhancement request: Ability to show the "has been cached" message multiple times on the same topic. Useful if you have a big topic where you want to give the user the opportunity to refresh the page in several places. Can be done with a new %VARCACHE{ showmsg="on" }% parameter, which only shows the message, but does not initiate a cache refresh action if the cache is outdated. If the user lists multiple VARCACHE on teh same topic, it must be ensured that the cache is only refreshed once.

-- PeterThoeny - 11 May 2007

A bit of feedback:

  • This plugin currently stores the cached topic in the pub directory. Should really be in the work area.
  • When it caches the TwistyPlugin (or any plugin that adds javascript or css files to the head), the imports are lost. Is there any way for the VarCachePlugin to know what needs to be added to the head and to add those even when the topic is cached? I suspect this wont be easy, but if anyone has any pointers then I don't mind looking into it. Maybe this could help: ProcessAddToHeadAdds

Thanks for the plugin.

-- AndrewRJones - 09 Aug 2007

The pub directory is the recommended storage area up to the Cairo release. This plugin currently runs on Cairo and TWiki 4.x. Long time passed since TWiki 4 release, so it makes sense to make this plugin require TWiki 4, and to move the plugin data to the plugin work area.

Ah, yes, javascript and css in head, that needs to be added. I ahve no time to look into this.

-- PeterThoeny - 12 Aug 2007

It would be nice if INCLUDEd page parts could be cached. So I can have a VARCACHE in the left bar to speed up all page views.

-- ArthurClemens - 13 Mar 2008

Good idea. There would be some limitations though, for example, the login user name would be cached. This could be addressed by specifying what should not be cached.

-- PeterThoeny - 13 Mar 2008

These comments apply to TWiki 4.2.3 and VarCachePlugin 2010-05-01.

I encountered a number of problems getting this plugin to work.

Here is the analysis & the patches to fix these issues.

This plugin should not be blindly setting umask - if it must set it, it should save and restore the previous value.

Despite the 2005 comment, cache files are being created with world read permissions. I see no permissions config paramenters, other than {RCS}{dirPermission} and {RCS}{filePermission}. Of course, TWiki::Func::saveFile doesn't respect these. Absent something better, umask should be saved/set to these/restored as appropriate. But moving the data files to working allows the usual sticky directory permissions to take care of this, leaving only the mkdir.

The test topic does not work on a new install because the topic cache directory structure does not exist, and won't be created without a refresh. The logic seems contorted: If ?varcache=refresh, we will create the cached version. Otherwise, we look for the absence of ?refresh in the URL, in which case we neither check NOR refresh the cache? This is actually trying to see if we have a non-?varcache parameter.

However, once this is fixed, the test topic still doesn't work if TablePlugin is installed because ?sortcol&table&up appear in $query->param() due to an over-aggressive restoration of these parameters. This is a TablePlugin bug.

--- lib/TWiki/Plugins/TablePlugin/Core.pm~      2008-09-11 23:41:58.000000000 -0400
+++ lib/TWiki/Plugins/TablePlugin/Core.pm       2011-05-18 21:38:14.000000000 -0400
@@ -1696,13 +1696,13 @@
         $url = $cgi->url(-absolute=>1, -path_info=>1) . '?';
         my $queryString = $cgi->query_string();
         $url .= $queryString . ';' if $queryString;

         # Restore parameters, so we don't interfere on the remaining execution
-        $cgi->param( -name => 'sortcol', -value => \@origSort );
-        $cgi->param( -name => 'table',   -value => \@origTable );
-        $cgi->param( -name => 'up',      -value => \@origUp );
+        $cgi->param( -name => 'sortcol', -value => \@origSort ) if( @origSort );
+        $cgi->param( -name => 'table',   -value => \@origTable ) if( @origTable );
+        $cgi->param( -name => 'up',      -value => \@origUp ) if( @origUp );

         $sortColFromUrl =
           $cgi->param('sortcol');    # zero based: 0 is first column
         $requestedTable = $cgi->param('table');
         $up             = $cgi->param('up');

The $paramMsg global with mod perl is fixable.

No-one ever moved the cache files to working - which actually simplifies the code.

$NO_PREFS_IN_TOPIC should be set for performance - after all, this is a performance-boosting plugin.

--- lib/TWiki/Plugins/VarCachePlugin.pm~        2011-04-30 02:05:45.000000000 -0400
+++ lib/TWiki/Plugins/VarCachePlugin.pm 2011-05-18 23:01:06.000000000 -0400
@@ -21,18 +21,20 @@
 package TWiki::Plugins::VarCachePlugin;

 # =========================
 use vars qw(
         $web $topic $user $installWeb $VERSION $RELEASE $pluginName
-        $debug $paramMsg
+        $debug $NO_PREFS_IN_TOPIC
     );

-$VERSION = '$Rev: 18518 (2011-04-29) $';
+$VERSION = '$Rev: 18518 (2011-05-18) $';
 $RELEASE = '2010-05-01';

 $pluginName = 'VarCachePlugin';  # Name of this Plugin

+$NO_PREFS_IN_TOPIC = 1;
+
 # =========================
 sub initPlugin
 {
     ( $topic, $web, $user, $installWeb ) = @_;

@@ -72,19 +74,20 @@

     TWiki::Func::writeDebug( "- ${pluginName}::afterCommonTagsHandler( $_[2].$_[1] )" ) if $debug;

     return unless( $_[0] =~ /%--VARCACHE\:/ );

-    if( $_[0] =~ /%--VARCACHE\:([a-z]+)\:?(.*?)--%/ ) {
+    if( $_[0] =~ /%--VARCACHE\:([a-z]+)\:?(.*?)(?:\001(.*?)\001)?--%/ ) {
         my $save = ( $1 eq "save" );
         my $age = $2 || 0;
+       my $tag = $3;
         my $cacheFilename = _cacheFileName( $_[2], $_[1], $save );

         if( $save ) {
             # update cache
             TWiki::Func::saveFile( $cacheFilename, $_[0] );
-            $msg = _formatMsg( $_[2], $_[1] );
+            $msg = _formatMsg( $_[2], $_[1], $tag );
             $_[0] =~ s/%--VARCACHE\:.*?--%/$msg/go;

             # cache addToHEAD info
             # FIXME: Enhance the TWiki::Func API to get the head info cleanly
             $cacheFilename = _cacheFileName( $_[2], $_[1], 0, 1 );
@@ -95,15 +98,14 @@
                 $htmlHeader = join( "\n",
                   map { '<!--VARCACHE_HEAD_'.$_.'-->'.$session->{_HTMLHEADERS}{$_} }
                   keys %{$session->{_HTMLHEADERS}} );
             }
             TWiki::Func::saveFile( $cacheFilename, $htmlHeader );
-
-        } else {
+       } else {
             # read cache
             my $text = TWiki::Func::readFile( $cacheFilename );
-            $msg = _formatMsg( $_[2], $_[1] );
+            $msg = _formatMsg( $_[2], $_[1], $tag );
             $msg =~ s/\$age/_formatAge($age)/geo;
             $text =~ s/%--VARCACHE.*?--%/$msg/go;
             $_[0] = $text;

             # restore addToHEAD info from cache
@@ -119,13 +121,12 @@
 }

 # =========================
 sub _formatMsg
 {
-    my ( $theWeb, $theTopic ) = @_;
+    my ( $theWeb, $theTopic, $msg ) = @_;

-    my $msg = $paramMsg; # FIXME: Global variable not reliable in mod_perl
     $msg =~ s|<nop>||go;
     $msg =~ s|\$link|%SCRIPTURL{view}%/%WEB%/%TOPIC%?varcache=refresh|go;
     $msg =~ s|%ATTACHURL%|%PUBURL%/$installWeb/$pluginName|go;
     $msg =~ s|%ATTACHURLPATH%|%PUBURLPATH%/$installWeb/$pluginName|go;
     $msg = TWiki::Func::expandCommonVariables( $msg, $theTopic, $theWeb );
@@ -166,14 +167,13 @@
     if( $query ) {
         my $tmp = $query->param( 'varcache' ) || "";
         if( $tmp eq "refresh" ) {
             $action = "refresh";
         } else {
-            $action = "" if( grep{ !/^refresh$/ } $query->param );
+            $action = "" if( grep{ !/^varcache$/ } $query->param );
         }
     }
-
     if( $action eq "check" ) {
         my $filename = _cacheFileName( $theWeb, $theTopic, 0 );
         if( -e $filename ) {
             my $now = time();
             my $cacheTime = (stat $filename)[9] || 10000000000;
@@ -185,27 +185,27 @@
                        || TWiki::Func::getPreferencesValue( "\U$pluginName\E_REFRESH" ) || 24;
             $refresh *= 3600;
             if( ( ( $refresh == 0 ) || ( $cacheTime >= $now - $refresh ) )
              && ( $cacheTime >= $topicTime ) ) {
                 # add marker for afterCommonTagsHandler to read cached file
-                $paramMsg = TWiki::Func::extractNameValuePair( $theArgs, "cachemsg" )
-                         || TWiki::Func::getPreferencesValue( "\U$pluginName\E_CACHEMSG" )
-                         || 'This topic was cached $age ago ([[$link][refresh]])';
+                my $paramMsg = TWiki::Func::extractNameValuePair( $theArgs, "cachemsg" )
+                           || TWiki::Func::getPreferencesValue( "\U$pluginName\E_CACHEMSG" )
+                            || 'This topic was cached $age ago ([[$link][refresh]])';
                 $cacheTime = sprintf( "%1.6f", ( $now - $cacheTime ) / 3600 );
-                return "%--VARCACHE\:read:$cacheTime--%";
+                return "%--VARCACHE\:read:$cacheTime\001$paramMsg\001--%";
             }
         }
         $action = "refresh";
     }

     if( $action eq "refresh" ) {
         # add marker for afterCommonTagsHandler to refresh cache file
-        $paramMsg = TWiki::Func::extractNameValuePair( $theArgs, "updatemsg" )
-                 || TWiki::Func::getPreferencesValue( "\U$pluginName\E_UPDATEMSG" )
-                 || 'This topic is now cached ([[$link][refresh]])';
+        my $paramMsg = TWiki::Func::extractNameValuePair( $theArgs, "updatemsg" )
+                    || TWiki::Func::getPreferencesValue( "\U$pluginName\E_UPDATEMSG" )
+                    || 'This topic is now cached ([[$link][refresh]])';

-        return "%--VARCACHE\:save--%";
+        return "%--VARCACHE\:save\001$paramMsg\001--%";
     }

     # else normal uncached processing
     return "";
 }
@@ -213,24 +213,17 @@
 # =========================
 sub _cacheFileName
 {
     my ( $web, $topic, $mkDir, $isHead ) = @_;

-    # Create web directory "pub/$web" if needed
-    my $dir = TWiki::Func::getPubDir() . "/$web";
+    my $dir = TWiki::Func::getWorkArea($pluginName) . "/$web";
     if( ( $mkDir ) && ( ! -e "$dir" ) ) {
-        umask( 002 );
-        mkdir( $dir, 0775 );
+        my $sumsk = umask( 002 );
+        mkdir( $dir, $TWiki::cfg{RCS}{dirPermission} );
+       umask( $sumsk );
     }
-    # Create topic directory "pub/$web/$topic" if needed
-    $dir .= "/$topic";
-    if( ( $mkDir ) && ( ! -e "$dir" ) ) {
-        umask( 002 );
-        mkdir( $dir, 0775 );
-    }
-    my $fileName = '_cache.txt';
-    $fileName    = '_cache.head' if( $isHead );
-    return "$dir/_${pluginName}${fileName}";
+
+    return "$dir/${topic}_cache." . ($isHead? 'head' : 'txt');
 }

 # =========================
 1;

-- TimotheLitt - 2011-05-19

Thank you Timothe for the patch. Yes it makes sense to move the cache into the working area.

I'll look into it later. For now I escaped the patch above, there was some interference by the VarCachePlugin running on twiki.org seeing its own code in a TWiki topic.

-- PeterThoeny - 2011-05-19

FYI, Foswiki has already moved the cache into the working area.

I subsequently noticed that

 is not accepted.  Simple fix:

@@ -57,11 +59,11 @@

     TWiki::Func::writeDebug( "- ${pluginName}::beforeCommonTagsHandler( $_[2].$_[1] )" ) if $debug;

     return unless( $_[0] =~ /%VARCACHE/ );

-    $_[0] =~ s/%VARCACHE{(.*?)}%/_handleVarCache( $_[2], $_[1], $1 )/ge;
+    $_[0] =~ s/%VARCACHE(?:{(.*?)})?%/_handleVarCache( $_[2], $_[1], $1 )/ge;

     # if "read cache", replace all text with marker, to be filled in afterCommonTagsHandler
     $_[0] =~ s/^.*(%--VARCACHE\:read\:.*?--%).*$/$1/os;
 }

-- TimotheLitt - 2011-05-19

Seem to have a streak running on this. One more. If refresh interval is defaulted to a web preference, zero (meaning infinite) can not be specified. In fact, if zero is explicitly specified as an argument, 24 hours will be used. Usual difference between undef and zero.

Line 183 (after the previous patches):

             my $refresh = TWiki::Func::extractNameValuePair( $theArgs )
-                       || TWiki::Func::extractNameValuePair( $theArgs, "refresh" )
-                       || TWiki::Func::getPreferencesValue( "\U$pluginName\E_REFRESH" ) || 24;
+                       || TWiki::Func::extractNameValuePair( $theArgs, "refresh" );
+            $refresh = TWiki::Func::getPreferencesValue( "\U$pluginName\E_REFRESH" ) unless( defined $refresh );
+           $refresh = 24 unless( defined $refresh );
             $refresh *= 3600;
-- TimotheLitt - 2011-05-20

This has become messy; I attached a consolidated patch to make it easier to apply. (I hope that's the end of this saga...)

-- TimotheLitt - 2011-05-22

Thanks Timothe, I'll test and take that into the next plugin release.

-- PeterThoeny - 2011-05-22

Thanks again Timothe for contributing this patch. This is now in SVN trunk and TWiki-5.0 branch. I did some minor fixes. For example, the extractNameValuePair returns an empty string, not undef, so the test needs to account for that.

-- PeterThoeny - 2011-05-24

Thanks. Have you tested your version with no preference value present?

I believe that in that case, getPreferencesValue will return undef, and your '= 24 unless( $refresh || $refresh eq '0' ) ' will return a 'undef used in comparison' error on the right side of the ||.

I think you want '= 24 unless( $refresh || defined $refresh && $refresh eq '0' ) '

Defaulting in perl is always trickier than it appears...

-- TimotheLitt - 2011-05-24

Good catch! Plugin updated and uploaded.

-- PeterThoeny - 2011-05-24

I would love to be able to INCLUDE VarCached data in another topic (specifically to include JSON-encoded data, for parsing with javascript). Do I have any options other than ajax-ing it in with a separate request?

-- AaronLWalker - 2012-08-17

When you INCLUDE another topic, that part and the content of the including topic gets cached. There is currently no way to INCLUDE text that is already cached.

May be you can use the AttachContentPlugin and include attached data. In a way this plugin produces cached data, although it does not auto-refresh.

-- PeterThoeny - 2012-08-17

I recently started using this plugin and found out that it makes the save operation slow on a web having thousands of topics. That's mentioned on VarCachePlugin, but that may not be enough. Because the plugin makes save slow regardless its usage. As such, on a TWiki installation having webs with many topics, TWikiPreferences should have:

   * Set VARCACHEPLUGIN_INVALIDATEONUPDATE = none
Alternatively, that configuration is regarded as "none" by default in VarCachePlugin.pm. What do you guys think?

-- Hideyo Imazu - 2017-01-12

Although I understand that the current default is a problem for large webs, I think that large webs are the exception, not the norm. How about leaving the default and setting the none just on large webs webs?

-- Peter Thoeny - 2017-01-12

If there are thousands of webs, you cannot expect all large web owners do that. We need to set VARCACHEPLUGIN_INVALIDATEONUPDATE = none as the site default. So I will mention that on VarCachePlugin.

-- Hideyo Imazu - 2017-01-12

A doc update looks like a good middle ground.

-- Peter Thoeny - 2017-01-12

I've come across a situation where only a subset of users are supposed to refresh the cache. This is because a) the topic in question has %SEARCH{...}%; b) SEARCH'ed topics have view restriction; c) only users who can view all of those SEARCH'ed topics can render the topic in question properly; d) the users who are supposed to view the topic in question are a superset of the users who are supposed to refresh it. To handle this situation, I'd propose to enhance the plugin to refer to the following refresh permission variables and act accordingly. What do you guys think?

   * Set DENYVCREFRESH = DeniedGroup, Main.DeniedUser
   * Set ALLOWVCREFRESH = AllowedGroup, Main.AllowedUser

-- Hideyo Imazu - 2019-12-12

I am fine with this enhancement. To be more specific and self explementory it might be better to name the settings DENYVARCACHEREFRESH and ALLOWVARCACHEREFRESH, respectively.

-- Peter Thoeny - 2019-12-13

OK. I'll use DENYVARCACHEREFRESH and ALLOWVARCACHEREFRESH.

-- Hideyo Imazu - 2019-12-16

FYI. I've filed TWikibug:Item7879 and will work on it.

-- Hideyo Imazu - 2020-01-22

Topic attachments
I Attachment History Action Size Date Who Comment
Unknown file formatpatch VarCachePlugin.patch r1 manage 7.2 K 2011-05-22 - 10:11 UnknownUser Consolidated patch for may 2011 issues
Unknown file formatdiff VarCachePlugin.pm.diff r1 manage 0.6 K 2006-03-06 - 10:39 UnknownUser fix for getting it to work with the 4.0.1 release
Edit | Attach | Watch | Print version | History: r45 < r44 < r43 < r42 < r41 | Backlinks | Raw View | Raw edit | More topic actions
Topic revision: r45 - 2020-01-22 - HideyoImazu
 
  • Learn about TWiki  
  • Download TWiki
This site is powered by the TWiki collaboration platform Powered by Perl Hosted by OICcam.com Ideas, requests, problems regarding TWiki? Send feedback. Ask community in the support forum.
Copyright © 1999-2024 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.