Feedback and discussion on TWikiCacheAddOn

-- RichardBaar - 27 Aug 2003

I installed the Add-on a couple of days ago. It seemed to work fine but one of my guys was unable to access one specific Web. He didn't actually have caching turned on but he couldn't get access to the web even when I turned caching on. It appeared to be a "permissions" issue but he has not problem when I replace the Cache Add-On version of ./bin/view with the previous version and has been accessing this same web daily for months.

I haven't been able to track down a possible cause but did see some error log output caused by your view code.

I did make an amendment to your view code as follows:

95c95
<     checkCacheDir( $cacheDir );
---
>     checkCacheDir( $cacheDir, "" );
178c178
<       checkCacheDir( $cacheDir );
---
>       checkCacheDir( $cacheDir , "" );

This stops errors from filling the webserver log because your sub checkCacheDir tries to test an uninitialized string variable. You also have errors on line 182 where you don't test the return value from a stat call before trying to use it.

I believe my previous version of view may well be a version which came with the Koala skin.

My question would be - would it not be more practical for your code to wrap around an installation's existing view ( e.g. rename the original view ) rather than completely replace it? It seems like the view you are replacing may have custom functions that will be lost by outright replacement.

-- EdMcGuigan - 04 Sep 2003

Hi,

well, our TWiki uses about 100 people and we have about 40 webs and this problem I never solve.
The only thing I did to original view script was adding my caching and authorization code I move from the end of the script to the beginning. In the original script the script at first renders the page and then it checks the permissions. To improve speed I did this check at the beginning and if the user pass than I render the page or send the cached page. In the algorithm I didn't make any changes. But there is possibility that I change in the past the Access.pm file and I forget about it.

The view script is not from KoalaSkin. It is from TWiki installation and Koala doesn't make any changes to it.

Could you, please, debug on which line the view script denies the access of the user?? Or you can try to move the authorization code back to the original place in the view script (but than the cache will not work correctly if you used one cache for all users - $usePerUserCache = 0).

-- RichardBaar - 09 Sep 2003

Hi Richard:

The problem appears to be due to a difference in the call to function userToWikiName in your view script. I turned on debugging in the Twiki::Access module and saw that the usernames were coming in as JohnDoe rather then Main.JohnDoe and this was the probable cause of permissions failure. The reason why permissions worked in the case of other users ( me ) is that I have some kind of superuser status.

When I grepped the view scripts for occurrences of the wikiUserName variable, I got:

bash-2.05$ grep wikiUserName view.cached 
    my $wikiUserName = &TWiki::userToWikiName( $userName, 1 );
    if ( $adminPerUserCache ) { $cacheDir .= "/$wikiUserName"; }
    my $viewAccessOK = &TWiki::Access::checkAccessPermission( "view", $wikiUserName, $text, $topic, $webName );
user : $wikiUserName test: $text topic: $topic web: $webName
bash-2.05$ grep wikiUserName view        
    my $wikiUserName = &TWiki::userToWikiName( $userName );
    my $viewAccessOK = &TWiki::Access::checkAccessPermission( "view", $wikiUserName, $text, $topic, $webName );

You can see that you have the additional parameter which when I checked in TWIki.pm has the effect of dropping the Main. from the front of the username.

I edited your view to remove that parameter and everything worked thereafter.

Let me know what you think.

-- EdMcGuigan - 09 Sep 2003

Hi Ed,

well, I have to say that this problem in our firm arises two hours later I read your comment. :-)) And I identify the same problem as you. But I solve it by creating a function in Func.pm that returns $wikiUserName always in the form $web.$topic. For me it was helpful because I need this function in more of my scripts. But I think that the result is in both solutions the same - IT WORKS!! :-)))

-- RichardBaar - 12 Sep 2003

All,

Let me start off by thanking RichardBaar for a great plugin. I recently installed Version 1.0 (plus changes suggested by EdMcGuigan above) and have noticed significant improvement in page load times as well as reduced CPU usage.

My main goal is to reduce CPU usage and load time of my WebRss topics. It appears that this plugin doesn't currently cache these pages though as they require a URL parameter (?skin=rss) to be viewed with RSS aggregators.

To work around this I added the following to the TWikiCacheAddOn view script. The effect is to make the rss skin the default for any topic which is named WebRss. This eliminates the need for the extra URL parameter thus allowing cache benefits to be realized for WebRss topics. I admit that this is a blatent hack, even so I believe it to be safe, effective, and convenient.

(Addendum) I've updated a number of things to enable more frequent invalidation of search pages when CACHESEARCH is on. Some of these are specific to WebRss. Below is a diff script which you can use to patch your TWikiCacheAddOn version 1.0 view script if you're interested in playing with this idea.

81c81
<     my $wikiUserName = &TWiki::userToWikiName( $userName, 1 );
---
>     my $wikiUserName = &TWiki::userToWikiName( $userName );
97a98,108
>     # Set the $skin to rss if the $topic is WebRss  -- JasonWhite 2/27/04
>     # FIXME: HACK! This is really just a hack. The cache plugin won't cache pages which 
>     #     have URL parameters passed to them. To use WebRss you need to pass ?skin=rss. 
>     #     So WebRss would never be cached which defeats the my whole purpose of 
>     #     installing the cache plugin. SO this gets around all of that and makes 
>     #     accessing the WebRss page more intuitive at the same time. 
>     #          --JasonWhite 2/27/04
>     if( $topic eq "WebRss" and $query->param( "skin" ) eq "" ) {
>       $skin = "rss";
>     }
> 
198,204c209,214
<           #include topic SEARCH??
<           if ( $text =~ /%SEARCH{.*?}%/ ) {
<               if ( $cacheSearch ne "on" ) { $readCache = 0; }
<           }
<           #include topic ACTIONSEARCH??
<           if ( $text =~ /%ACTIONSEARCH{.*?}%/ ) {
<               if ( $cacheSearch ne "on" ) { $readCache = 0; }
---
>           #FIXME: This works for directories on my Win & Linux boxes but may not work for database users. Investigate. -- JasonWhite 2/28/04
>           my @webStat = stat( "$TWiki::dataDir/$webName" );
>           
>           #include topic (ACTION)SEARCH?? Also invalidate cache if the topic web has been updated AT ALL.
>           if ( $text =~ /%(ACTION)?SEARCH{.*?}%/ ) {
>               if ( $cacheSearch ne "on" or $webStat[9] > $cacheStat[9]) { $readCache = 0; }              
205a216,222
>           
>           #FIXME: HACK! If cached WebRss page is older than last web update don't use the cached version.
>           #   (See "...%INCLUDE{*}% topics containing %SEARCH{*}%..." FIXME comment below for a better solution.) -- JasonWhite 2/28/04
>           if ( $topic eq "WebRss" ) {
>               if ( $cacheSearch ne "on" or $webStat[9] > $cacheStat[9]) { $readCache = 0; }              
>           }     
>           
210a228,230
>           #
>           # FIXME: Topics which <nop>%INCLUDE{*}% topics containing %SEARCH{*}% tags are incorrectly cached.
>           #

Other solutions to consider:

• Adding functionality for TWiki variables to WebPrefrences which allow you to specify default skins for topics.
• Adding (smart) support for skin and/or URL parameter caching to the TWikiCacheAddOn.
• Add support for topic variables which specify topic cache parameters. This seems much more manageable than expanding the logic of view cache control to accommodate every possible scenario. For Example:
  • %CACHE{ maxAge="2days" }% could specify that any cached version of this page over two days old should not be used. A setting of hours or minutes would allow most search pages to stay current while still allowing for cache benefits on high traffic sites.
  • %CACHE{ newerThan="Plugins/WebPreferences"} could explicitly define dependencies. In this example we'd only use the cached version of the page if it was newer than Plugins/WebPreferences.
• I've also been playing with using gzip to compress cached pages. This saves disk space but my real motivation is to serve the compressed pages. Most browsers will deal with compressed pages correctly: it's part of the HTTP 1.1 recommendation. Most pages compress to a tenth the size which reduces bandwidth requirements and significantly reduces download times especially for modem users. For example a WebChanges I tested was 37472 bytes and compressed to 4423 bytes this could mean a difference of 7 seconds in download time! Web page compression is typically implemented as a part of caching solutions, as the common spirit is to reduce resource usage, so I think this functionality is a good fit for the TWikiCacheAddOn. To get a feel for how this would effect your pages try a few samples in this class.gzip_encode.php demonstration page. I wouldn't be using PHP but I would use zlib so the CPU times and compression results would be identical.

-- JasonWhite - 28 Feb 2004

---

I tried to install this plugins, that looks great, but unfortunately failed to have it to work.

I am using cairo with PatternSkin. Fist I had to hack a bit the script to replace a few TWiki::getRenderedVersion with TWiki::Render::getRenderedVersion (attached). then I added

$cacheDir       = "/twiki/cache" ;
$useCache       = 1 ;
$usePerUserCache = 1; 

to Twiki.cfg, and then Set USECACHE = on to TWikiPreferences.

results gives no errors, but "/twiki/cache" stays empty, and I see no perf changes.

did I miss anything ?

* view: hacked view with ::Render for cairo

-- LucAbom - 29 Dec 2004

my apologies. installed it on another web, and it did worked. ( well, using the modified scipt). only difference I can see is that I did not create the cache dir by hand, but let the script doing it. could it be some access issues ?

-- LucAbom - 17 Jan 2005

attached is a script hacked for looking if %INCLUDE% contains , and do not cache them ( which is the case in cairo/ pattern, where the WebChanges is an include of =TWiki06x01.WebChanges ). also my previous scipt introduced a bug in L208 , so included where non cached anyway.

• view: hacked for INCLUDE with SEARCH not to cache (WebChanges)

-- LucAbom - 19 Jan 2005

I cannot seem to get this to work for the life of me on the latest stable rel. I think I have tried most everything I can think of with no luck. Has anyone gotten this plugin to work with the latest stable cairo release?

-- JamesMalone - 26 Jan 20

I'm also running into problems with this script using FC2 and Apache 2 and I'm not quite sure what to try to resolve the problem. I start with a functioning TWiki20040902 using the supplied view.pl script and rendering pages just fine.

• I've made the changes to TWiki.cfg as suggested - specifically: $cacheDir = "/home/twiki/cache";
• When I replace the view program with the one from TWikiCacheAddOn.zip (27 Aug 03) I get Software error: Undefined subroutine &TWiki::getRenderedVersion called at /home/twiki/bin/view line 274.
• If I apply the username hack from JasonWhite (above) I get the same error message
• Replacing the view script with the LucAbom hacked script gives the following error: Internal Server Error The server encountered an internal error or misconfiguration and was unable to complete your request.

Anyone else able to get this working with FC2? Would it be better to spend time getting SpeedyCGI working instead?

-- TomKusleika - 24 Mar 2005

Congradulations for this great addon. Actually, I've been hacking some very similar code to get our site responsive on the base of an ancient beijing installation. I had a lengthy email discussion with AntonAylward about caching and he is actually not a fan of caching any render output of TWiki (sorry, Anton, for giving a very shortended summary of our chat but let me continue).

Here are some points he made that might be of interest for this untertaking: caching html pages is a good solution for beijing and cairo which both fail in performance due to intensive IO inside TWiki and recursive topic rereading (getting variables, checking access permissions etc). So technically caching html pages is a good solution to those TWiki engines. But things changed in dakar: IO has been reduced massively; information isn't extracted twice from a topic; the ratio between perl-compl and perl-exec has been improved albeit dakar has a lot more code; the perl-exec path has been shortened to finish the cgi; TWiki is completly object-oriented and thereby targeted to run in mod_perl zeroing perl-compl.

This is all vergy good news for all of us, using a TWikiCacheAddOn or not!!!

But here are some of my points: although perl-exec inside the TWiki engine is being optimized in dakar, external processes (grep for SEARCH, latex for MathModePlugin, bib2bib+bibtool+bibtex+latex2html for the upcoming BibtexPlugin and others) that get forked and who's output is then sucked in. Some of these plugins have their own solution of its own. Even the beloved HeadlinesPlugin prevents extensive site-polling.

So there are two solutions: (a) either use some CacheContrib for all of those plugins that providing the needed caching or (b) address the caching thingy twiki-wide as this addon does. (a) has the advantage that it is a SmallGun (tm) and is reasonable code-sharing between plugins. (a) has the disadvantage that it does not address SEARCH being no plugin. (b) has the big disadvantage that it doing caching right is a BigGun (tm) and the algorithm used in the TWikiCacheAddOn needs some rethinking as I will elaborate below. (b) has the big advantage of at least keeping TWiki's rendering time (view) constant, no matter how many plugins you install. (b) is reasonable even for site with much configurability per user and alot of other dependencies for cache entries.

Actually, why shouldn't we do both, I'd say.

And now some more remarks on the current algorithm of this addon compared to some of my own unpublished code:

The key to caching is dependencies.

• A cache entry is only valid as long as none of its dependencies has fired.
• Dependencies must be tracked for each topic collecting information about what was relevant for rendering the view.
• The validity of Cache entries shouldn't be checked on view time as this is the code passage that should be as lean as possible.
• Cache entries should be invalidated during save, but therefor the reverse dependencies are needed (that is: "who is depending on me" vs. "what do I depend on"). save will look up all rev-deps and delete all topics in all user-caches that they point to.
• Dependencies are of different type that induce more or less cache invalidation:
  1. plain file modification time: someone tinkers with .txt files outside the scope of TWiki controlling topic changes
  2. simple link: rename will change the link appearance in other topics
  3. simple include plus simple link in included topic
  4. search: very unpredictable and fired when any topic in the scope of the search is altered (see the web argument of search)
  5. web preferences: all topics of the same web depend on the WebPreferences, but also on the WebMenu, WebLeftBar and what ever
  6. site preferences: all topics in all webs depend on them (might be ignored)
  7. user preferences: all topics in all webs depend on them
  8. template: all topics depend on the used templages (might be ignored)
  9. TWiki engine: all topics depend on TWiki.pm etc (might be ignored)
  10. custom: one might establish or release dependencies by hand per topic using something like %ADDDEP{"topic-regexps"}% and %DELDEP{"topic-regexps"}%
  11. plugin: plugin authos might use a dependency API in order to establish dependencies for a topic using %FUNKYTAG%
• Dependencies might be of interest: let's have a %RENDERDEPS{topic="" format=""}% and %RENDERREVDEPS{...}%
• Dependencies might be invalidated by hand, either only one or all in a web or all of the site
• Caching a topic might be prevented per topic, web or site (already implemented here, I think)

Well, that's what I wanted to say for a long time. And as you see this realy is a BigGun (tm). Maybe it is a good idea to just make a DependencyTrackerAddOn to get these ideas tested.

Last not least, any caching solution will have to face dakar, that is improved performance without any caching (I'm already working with DEVELOP on my localhost. Wow what a difference!), dakar is realy a big change to the TWiki engine. Plugins and addons should be made dakar-ready.

-- MichaelDaum - 25 Mar 2005

Michael wasn't really selling me short, and his summary above is excellent. The improvements in Dakar for an "_out of the box_" installation or ones without those heavy hitting plugins is great.

But I do favour caches. Just not on the server.

As has been pointed out elsewhere, not least of all in this forum by people such as CrawfordCurrie, the user's exerience and how fast the perl code rneders and emits the HTML of a topic are two quite different things.

Lets assume that the perl itnerpreter is infinitely fast. OK, its a fantasy, but bear with me. Even a 100% improvement in rendering speed makes this discussion valid. As Michael touched on, the server still has to pull stuff out of the file system, and each topic needs to have the preferences checked the User, Web and TWiki levels. All the plugins need to pull their configuration pages (and argument for separating plugin documentation and configuration so the configuration pages are shorter and faster to read). There is the skin template and and left, top and bootom bars. That's just for starters.

This gets cached, but it is the OS file system chache. As TWiki developers we have no cotnrol over it; as TWiki site admins we may or may not, but with modern OS the best we can do is make sure our boxes have adequate memory. Virtual memory, as the old saying goes, means virtual performance.

But when the HTML gets to the end user's browser it has to be rendered into an image. To do this, it will have to pull the CSS file and any javascript files. No amount of improvement in the rendering process will help here. It doens't matter if the perl interpreter is infinitely fast or the HTML cache on the TWiki server is perfect, the browser still has to pull these files from the server - or wherever. (It may not be the same machine, but it usually is.)

Some plugins require you to modify the templates so that any view will include a reference to a Javascript file, wehther it is being used or not. This adds overhead to every view. Its why I object to this practice, but that's another matter.

Now back a few lines I said that I do use a cache. Its called Squid, and it sits between my browser and the network. Its set up so that it does cache the CSS and javascript files. Yes, my browser still has to pull them, but the response is a lot better than pulling them from TWiki.org.

Caches are not magic solutions. They have to be used intelligently and appropriately. Sometimes a cache is not the best, correct, or even appropriate solution. The Dakar rendering engine is showing that sometimes re-writing the code to reduce unnecessary IO is a better, more pervasive solution.

Or of course you could give up CSS skins and javascript, and plugins that take time to fetch, compile and run; that would improve performance as well :-/

Might I also re-echo Michael's parting comment. Could plugin developers please revisit their code and work to make it Dakar ready. Performance always sells and the core performance improvement of Dakar is substantial. If it can impress Michael, who avoided Cario because of the performance loss there, it must be good! Kudos to CC and others. But please, plugin devvelopers, don't let people be put off upgrading to Dakar because of incompatible plugins.

-- AntonAylward - 25 Mar 2005

Anton - you are stressing a point that every browser-based content management system has to face: getting the **king stuff transmitted to the user. So TWiki should be as slow fast as others. Not everybody will install a squid additionally to his/her browser.

The thing that has been tackled in the TWikiCacheAddOn and been followed up in this discussion shall not address the slowness of browsers and the internet per se. We cannot do anything about that besides telling everybody "get a faster computer with lots of (virtual) memory and, yes, dump your 28K modem" ...

Let's get TWiki as fast as a CMS can be.

-- MichaelDaum - 25 Mar 2005

All in all, a fascinating dissertation on the merits and tradeoffs being considered for the dakar release, but... Was there an intent (somewhere) to address my posted question about the failure of the CacheAddOn to run and/or how I could find and fix the problem? (I am reminded of the hoary helicopter-in-the-fog joke vis Microsoft)

-- TomKusleika - 25 Mar 2005

Tom, would you report this as a bug, plese, and in doing so specify your system details, perl and apache versions,

On my platforms TWiki::getRenderedVersion() is in lib/TWiki.pm in the version dated 13 May 2003, but is in lib/Twiki/Render.pm in the version dated 1 Sept 200 (1742). These are roghly Bejing and Cairo respectively. You seem to have a view from the former and code from the latter.

The above took me a couple of seconds with grep. Perhaps because grepping for this was so simple, and perhaps because this was marked as 'PluginDevelopment' rather than a bug report, Michael and myself didn't see it as a bug.

Not least of all since solving similar problems by using grep was mentioned in this topic on 09 Sep 2003.

Please report bugs using the bug report form. This topic is for discussion the development of the add-in, which is what Michael and I were doing.

-- AntonAylward - 28 Mar 2005

firstly - I'd like to say thankyou , and well done. we're using this AddOn on twiki.org at the moment, and its made the slow host useable for the interim!

I've found a Bug

1. edit a page, and create a link to a non-existant topic
2. create the new page from that link
3. some user views the page from step 1
4. you save the new page (for the first time)
5. now the page from step 1 looks like the topic does not exist.

cool huh - the fix will have to involve some tracking of topic creation

-- SvenDowideit - 22 May 2005

I don't know whether it is the cache add on or something else, but I notice that since the changeover there are many situations where the diff to the previous version just shows an empty diff, but it appears that something has changed on the topic. Maybe there are some strange interactions between diff and the cache plugin?

-- ThomasWeigert - 26 May 2005

Thank you thank you RichardBaar, this Add-on is currently a life saver for twiki.org while we wait for the new hardware

I found some issues I'd like to report:

1. WebChanges of Plugins and Support show outdated changes. This is probably because those pages INCLUDE another pages that does a SEARCH. (LucAbom provided a fix)
2. If you create a new topic by clicking on the questionmark, then go back to the parent topic and reload that, you still get a questionmark link. Very confusing for first time users.
3. Cache add-on does two log entries if topic is cached. This scews the statistics.

-- PeterThoeny - 09 Jun 2005

I discovered a limitation: This Add-on is not compatible with the BlackListPlugin. If a TWikiGuest is accessing an uncached topic it will be cached with a "You are black listed" message. Any other guest looking at this topic will see the back list message instead of the topic content.

-- PeterThoeny - 14 Jun 2005

checked .zip into CVS

-- WillNorris - 19 Jul 2005

Hey guys,

I am not sure whether it is something weird I am doing, such as running the view script from the command line using Perl, though when I do use perl to run the view script (perl view) I get a bunch of warnings from Perl:

bash-2.05a$ cd public_html/tswikidev/bin/
bash-2.05a$ perl view
[Fri Aug 12 01:33:57 2005] view: Name "TWiki::cacheGroup" used only once: possible typo at view line 57.
[Fri Aug 12 01:33:57 2005] view: Name "TWiki::cacheFileRights" used only once: possible typo at view line 55.
[Fri Aug 12 01:33:57 2005] view: Name "TWiki::cacheDirRights" used only once: possible typo at view line 56.
[Fri Aug 12 01:33:57 2005] view: Name "TWiki::usePerUserCache" used only once: possible typo at view line 54.
[Fri Aug 12 01:33:58 2005] view: Use of uninitialized value in string ne at view line 464.
[Fri Aug 12 01:33:58 2005] view: Use of uninitialized value in string ne at view line 464.
[Fri Aug 12 01:33:58 2005] view: Use of uninitialized value in string ne at view line 241.

Anyone else have this problem? Is there a reason for this behaviour? It does not look serious, though it caught my attention so I figured I would ask.

-- JamesMalone - 12 Aug 2005

Anyone tried this with Dakar?

-- MartinCleaver - 23 May 2006

I added a SHORTDESCRIPTION to the "Add-On Info" section so that this add-on is represented properly in the AddOnPackage topic and query topics. Please feel free to take this into the next release.

-- PeterThoeny - 04 Nov 2006

The view script seems to be based on an old pre-OO version of TWiki (ie. before view became just TWiki::UI::run ...). I assume this shouldn't be used with any modern version of TWiki such as Edinburgh release?

-- CallumGibson - 09 Mar 2007

There's ongoing work at TWikiCache to integrate a html page cache as well as a general caching service to the TWiki core in one of the next TWiki releases. This will make this addon package unnecessary.

-- MichaelDaum - 09 Mar 2007

After installing the plug-in as suggested, I get the following error:

Software Error: "Can't use string ("SKIN") as a HASH ref while "strict refs" in use at /opt/coolstack/apache2/htdocs/twiki/lib/TWiki/Prefs.pm line 254."

Any suggestions?

-- PrateekParekh - 09 May 2008