create new tag
, view all tags

PublicCacheAddOnDev Discussion: Page for developer collaboration, enhancement requests, patches and improved versions on PublicCacheAddOn contributed by the TWikiCommunity.
• Please let us know what you think of this extension.
• For support, check the existing questions, or ask a new support question in the Support web!
• Please report bugs below

PublicCacheAddOn Discussions

Alas, sending the page on the network is only one part of total time of getting a page on our browser. Now we need to speed up the other parts, by working on a "speed skin" minimizing the calls to other URLs (images, css, javascript), maybe "compiling" them (see http://developer.yahoo.com/performance/rules.html ). To get a glimpse of what is possible, just try to browse a cached TWiki site with Dillo, which ignores all the fancy CSS and javascript. It is really a tremendous experience!

Also, one of the tricks to survive spikes is to be sure you do not try to build the same page twice at the same time. I do this by locking (on unix, mkdir and ln -s are the two atomic operations allowing you to lock simply and reliably), and waiting random times to check the lock to avoid all the processes to wake up at the same time. Of course, shit happens so after some time the lock breaks, and you have to be careful to not mess up things if the process whose lock was broken wakes up afterwards. I think I got things right, but it is definitely tricky. But in my tests, locking was the mandatory feature to be able to send pages faster that a "while true;do wget" on a machine 3 time faster on a LAN could churn out

-- ColasNahaboo - 13 Jan 2008

Thank you Colas for contributing this add-on, very useful for public TWikis! Impressive speed! Overall good documentation. Small feedback:

  • avoid direct links to TWiki.org, use Interwiki links
  • escape the URL link in the "test on LAN" line
  • add also a zip file (expected by the query pages)
  • I tagged the add-on topic; please help tagging content

-- PeterThoeny - 14 Jan 2008

Colas, you doubled the efforts in TWikiCache.

-- MichaelDaum - 14 Jan 2008

(See the discussion in TWikiCache). I think I will re-orient my strategies to try different approches from it. For instance, I guess edits in a public site are done many at a time, but only some times per day. Even here on twiki.org it seems about one hour between edits. One strategy thus could be, when an edit is made, make the front end let go through all requests from the author IP, so he get to see perfect fresh copies of the site while he is editing, while the rest of the world see the cached copies (and do not put load on the server, so editing will still be faster than without cache). And then, clear the cache after he hasn't edited the site for N minutes (15? 60?). This way you get performance and accuracy for the editor, with the added bonus of "publish workflow": you can edit topics at will, you known you have one hour to correct things, and that you can save your work in mid-edit without it being seen in this half-ready state. The drawback is that it will not work well for collaborative sites (unless you know the IP of your co-workers and edits disable the cache for the team, not just dor you.). We can also bypass the cache based on session info... Anyways, there seem there are really many leads to follow!

-- ColasNahaboo - 15 Jan 2008

Ok, I have changed the implementation:

  • configs are autodetected, so there should be less problems like the ones Guilherme has encountered
  • a different approach have been taken from TWikiCache: cache cleaning strategy is now based on the properties of the editor (IP), not the page itself
  • I have installed it as a demo on a test machine at http://wiki.koalaz.net
  • it is now much more tested

-- ColasNahaboo - 29 Jan 2008

I have fixed it so no it runs on old versions too: Cairo, Dakar, Edinburgh... basically it only needs a afterSaveHandler in the plugin interface. Bumped the version num to 3 to indicate this change. I think all the needed functionalities are here now, so do not plan to enhance it in the near term, except for bug fixes of course.

-- ColasNahaboo - 03 Feb 2008

(non relevant discussions cleaned - you can see them at rev17

-- ColasNahaboo - 05 Feb 2008

with the 3.4 version, I now think that this cache system is of production quality

-- ColasNahaboo - 28 Feb 2008

Users of this plugin may want to try the FirefoxBoosterPlugin that can offer an interesting complement for some public sites.

-- ColasNahaboo - 08 Mar 2008

Addon now also in SVN (See Bugs:Item5424)

-- ColasNahaboo - 08 Mar 2008

Note that although I have set the modification policy to ContactAuthorFirst, I do not actually request you contact me first but just check the current dev version before, that is on http://hg.colas.nahaboo.net/twiki-colas/twpc/ to see what I am up to, and to ease merging your modifications with my dev version.

-- ColasNahaboo - 09 Mar 2008

Is it required to add the tag PCACHEEXPTIME to a topic to cache it? Or does it only override the expiration time?

  • CN: No, all topics are cached forever by default (until next edit of any other topic, that is). PCACHEEXPTIME should be used on topics that change by external causes without any user editing, e.g. a topic showing the time since last edit. Note that INCLUDE of external urls already calls PCACHEEXPTIME.

-- ArthurClemens - 09 Mar 2008

I am having an issue with a special character. The uncached topic shows En, but the cached version shows E?n. Is there any setting I should change?

  • CN: arf, I guess it is a bug. with the method I use (wget) I do not see the encoding/charset of the page, so the page is saved in the cache without any HTTP header info (neither content-type, charset, last-modified, expire, ...) , and given back with the default apache setting. Try to see if the encoding of the non-cached and the cached version differ. If so, try to set apache default encoding for the TWiki pages to the ones matching your TWiki install. I will need to find a way to also save the HTTP headers in the cache. A quick easy workaround for me could be also to add options to twpc to set these HTTP options.

-- ArthurClemens - 10 Mar 2008

After a blazing start this morning I have uninstalled the addon because each page request creates a number of processes that take a lot of cpu. I have looked at the C compiler option, but I am not sure what is needed for that.

-- ArthurClemens - 10 Mar 2008

Mmm this is strange. reducing CPU is the goal of this addon, so something is definitely not normal. Even without the C compiler option, it should be quite light on ressources (unless you requeted background builds?). For compiling, it needs just a C compiler. If something is wrong the installer should detect the failure and install the shell version instead anyways. I am curious of seeing the processes that were taking the CPU.

-- ColasNahaboo - 10 Mar 2008

It may have nothing to do with the addon. I was getting lots of entries in the log like: - - [10/Mar/2008:11:10:03 +0100] "GET /twiki/pub/TWiki/TagMePlugin/tag_addnew.gif HTTP/1.1" 200 297 "http://twiki/twiki/bin/view/Main/WebSearch?search=or&scope=all&web=Main" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; .NET CLR 1.1.4322; InfoPath.1)"

I am investigating the issue.

  • FYI, hits to build a cache are done with wget, so you can see it in the log as the user agent (and - as referer), as in:
    "GET /bin/vief/TWiki/TWikiPreferences HTTP/1.0" 200 200788 "-" "Wget/1.10.2"

-- ArthurClemens - 10 Mar 2008

Still not sure what it is. But with top I am getting CPU of 60% or higher when a topic has not been cached. So multiple hits can easily drown the server.

One bug: anchor links get the script vief instead of view in the link. It causes the page to reload (and recache?).

  • CN: arf, I fixed this early on... it has crept back (it will only reload, not recache). Normally I perform a search/replace of vief to view (and /bin/view to shorter form if set in TWiki config) when saving the cached file. Thanks for all your testing, btw.

-- ArthurClemens - 10 Mar 2008

I believe my webserver is not set up properly. Normal view requests eat up 60% of CPU. I first have to get that down (how?) before using the addon on that server.

-- ArthurClemens - 10 Mar 2008

I have set up mod_perl now. But then I get these errors:

Can't modify constant item in scalar assignment at /var/www/html/twiki/bin/pcad line 7, at EOF\nString found where operator expected at /var/www/html/twiki/bin/pcad line 7, near "case "$HTTP_ACCEPT_ENCODING""\nsyntax error at /var/www/html/twiki/bin/pcad line 7, near "case "$HTTP_ACCEPT_ENCODING""\nBareword found where operator expected at /var/www/html/twiki/bin/pcad line 7, near ""$HTTP_ACCEPT_ENCODING" in"\nBareword found where operator expected at /var/www/html/twiki/bin/pcad line 7, near ")gzip"\nBareword found where operator expected at /var/www/html/twiki/bin/pcad line 9, near "/}\nbin=/twiki"\n  (Might be a runaway multi-line // string starting on line 8)\nsyntax error at /var/www/html/twiki/bin/pcad line 9, near "/}\nbin=/twiki"\nBareword found where operator expected at /var/www/html/twiki/bin/pcad line 10, near "/twiki/bin"\nBareword found where operator expected at /var/www/html/twiki/bin/pcad line 11, near "/var/www"\nBareword found where operator expected at /var/www/html/twiki/bin/pcad line 12, near "/var/www"\n/var/www/html/twiki/bin/pcad has too many errors.\n
[Thu Mar 13 15:05:53 2008] [error] Unrecognized character \\x7F at /var/www/html/twiki/bin/view line 1.\n

I have followed the documentation to exclude view from mod_perl. Inside

<Directory "/var/www/html/twiki/bin">
... I have:
<FilesMatch "^(?!view|configure)[a-z.]+$">
   SetHandler perl-script
   PerlResponseHandler ModPerl::Registry
   PerlSendHeader On
   PerlOptions +ParseHeaders

-- ArthurClemens - 13 Mar 2008

  • I uploaded the 3.6 version that should fix the content-type & encoding problem
  • The vief problem in %TOC actually only happen while you are a "changer" or "bypasser" temporary state, it do not appear for normal readers or once the cache is reset
  • Your FileMatch line is bogus and match any script. You should have written it <FilesMatch "^(vief||viewfile|edit|preview|save|attach|upload|rename|installpasswd|passwd|manage)$">

-- ColasNahaboo - 16 Mar 2008

Thanks, that works a lot better!

-- ArthurClemens - 16 Mar 2008

Clearing the cache per topic from the PCAD menu does not seem to work. Using URL params cleartopics and topiclist I do get the correct result with the feedback "clearing cache: Done". Using the button "clear topics" gives the feedback "clearing cache of --" and no effect on the page.

Second, it would be great if I could create a "clear cache" link on pages with a "redirect" parameter back to the current topic. Fir instance> [[%SCRIPTURL{pcad}%/?cleartopics=on&topiclist=%WEB%.%TOPIC%;redirectto=%WEB%.%TOPIC%][clear cache]].

The documentation says http://twiki.org/cgi-bin/pcad?build build caches for all yet-uncached pages. Unfortunately this clears the cache of all topics first so it needs to cache everything all over.

-- ArthurClemens - 16 Mar 2008

  • cache: ah, it is a bug. The menu handling is too simplistic now, I will have to refactor if for next version
  • good idea. I have put on my TODO list for next version.
  • I could not reproduce it. Are you sure it was not caused by other things? (a reinstall, editing a page, ...)

-- ColasNahaboo - 17 Mar 2008

Today I encountered a nasty bug. The cache changes the action url of a html form, from:


So it doubles Web.Topic. This is a form created with FormPlugin.

-- ArthurClemens - 18 Mar 2008

On doubling: strange. Could you mail me the contents of your bin/pc-config ?

-- ColasNahaboo - 19 Mar 2008

Ok, uploaded 3.7, with bug fixes and the redirectto implemented (with a syntax slightly different from yours for more versatility).

On the "doubling" web.topic, it seems to me a bug of the FormPlugin in 4.2, as I have the bug even if the Cache addon is not installed... (an incompatibility with short urls?)

-- ColasNahaboo - 22 Mar 2008

New version v4.0 uploaded, with an incompatibility: cache control directives are now done via setting a preference variable PUBLIC_CACHE_EXPIRE instead of using a TWiki tag %PCACHEEXPTIME% in order to benefit from the scoping of TWiki vars (be able to set a cache policy on a whole web). To upgrade to this version you must:

  • uninstall then reinstall (although the install script should do it automatically)
  • change all the PCACHEEXPTIME you may have used in your topics to declarations like * Set PUBLIC_CACHE_EXPIRE = 1

-- ColasNahaboo - 23 Mar 2008

New version 4.1 uploaded, small changes in documentation and stats output.

-- ColasNahaboo - 05 Apr 2008

Nice to see progress in TWiki's caching technology!

-- PeterThoeny - 06 Apr 2008

I forgot to track changes made by plugins (via the TWiki::Func::saveFile), fixed in 4.2. upgrade recommended if you use tags.

-- ColasNahaboo - 08 Apr 2008

4.3 version now generates ETags on cached pages, to avoid re-serving them

-- ColasNahaboo - 09 Apr 2008

I am not sure if this is already implemented and I am missing some setting. But I like the way VarCachePlugin caches pages on view (if the cache is outdated), not just on edit.

-- ArthurClemens - 10 Apr 2008

I tried to use this AddOn on a hosted environment, where I'm forced to use *.pl filenames for perl scripts. For TWiki standards I can set this using the configure script and renaming the script manually. Do you intend to make this nice AddOn aware of the script extension setting in the LocalSite.cfg?

-- RetoGantenbein - 10 Apr 2008

Arthur: the cache is built on views, not edits. Saves just marks the cache as obsolete, but the cache generation will only happen on-demand, on next view. Maybe the pcad build command (or pcge) confused you: pre-building the cache is not necessary, these are just convenience functions.

Reto, thanks for the input. I will definitely implement this for next version!

-- ColasNahaboo - 10 Apr 2008

But isn't it strange I only see new cached files in working/public_cache when a topic has been edited, not when it has been viewed?

-- ArthurClemens - 10 Apr 2008

Reto: try the new 4.4 version that adds support of ScriptSuffix (and only this so other people do not need upgrading). Note that it vill only add the suffix to the perl script vief. If your setup requires a .sh suffix for shell (bash) CGI programs, please do a symbolic link view.sh towards view and declare .sh as the new prefix if you use the default mode. But I do not know what/if you need a prefix for compiled C, in case you want to use -O ...

Arthur: it works this way: default config (or -tX with X not 0):

  • view: page cache is built if it didnt exist, and served
  • edit: nothing is done
  • save: cache is not changed, but your IP is declared as a file in working/public_cache/cache/_changers/ From now on until some grace delay (17mn by default), you will bypass the cache, and your views do not trigger building the cache, although you can see the old cached page in the cache dirs that will be served to people other than you, saving CPU for you the editor, so that in practice this cache will make your edits faster if visitors browse your site while you edit. This is most likely the situation you are seeing. 17mn after your last edit (the -t parameter) teh cache will be cleared, as your "changer" status. Note that this allow you an automatic "draft mode", your article will only be "published" after 17mn of inactivity, your visitors seeing an old, but consistent, view of your site in the meantime.
with -t0 (synchronous mode), as above, except
  • save will just clear the whole cache (actually moving it to /working/public_cache/cache.NNN, a "trashcan" state before actual deletion) . note that the view that immediately follows the save will trigger the building of the cache of the page. This is what you are seeing?

Note: the last planned enhancement I want to do is to still build cache pages even for changers, so that their browsing just after editing will be accelerated and serve to prepare fresh cache pages to populate the cache on the impending clear.

-- ColasNahaboo - 10 Apr 2008

Thanks for the explanation. I will try again.

Only... The latest version strips /bin from links.

-- ArthurClemens - 10 Apr 2008

Opps Arthur, I forgot to test on non-ShortURLs installs :-(. v4.5 fixes this

-- ColasNahaboo - 11 Apr 2008

Yes, but now it renders links as http://company.com/twiki/twiki/bin/view/Main/WebHome (twice twiki/).

-- ArthurClemens - 11 Apr 2008

Arg... to much precipitation. v4.6 should be ok now.

-- ColasNahaboo - 12 Apr 2008


We are using TWiki internally behind a firewall. There are several internal proxies which makes the IP address NOT unique to identify changers for edits. We do use a NTLM based authentication which means each user has a unique username. Could this information be used to uniquely identify the changers?

By the way: I really like the clean look you are using on your personal site smile

-- CraigMeyer - 16 Apr 2008

Craig: If N people share the same IP, and one of them edit a page, all of them will become changers, and bypass the cache. This will just slow down things but will not introduce problems (each edits will still be correctly attributed to each people) So I guess it depends on the actual number of people under the same IP: If they are few, this should not be a big problem. If they are many (at my company, some years ago, the whole R&D shared the same IP...) you will be better off using Synchronous mode ( -t0 ).

On NTLM, I could use it if I have access to the information in the view mode (environment variable ?) to get user login. Note that I only need this for views after an edit, anonymous browsing before is OK. But I do not know NTLM, so please give me pointer to the configuration you use.

On the looks, thanks. I am still developing it and will release it when I am satisfied with it, but it consists only on small customizations of the Pattern Skin.

-- ColasNahaboo - 16 Apr 2008

On the I18N issue reported in March by Arthur: HTTP headers of the TWiki generated pages are importan, as they include a character encoding header - wget does have an option to include headers but you'd have to do make sure that just the Charset header is sent when you server up a cached page. I'm mentioning this just for completeness...

A better and simpler approach is probably to ensure that the Apache configuration for default charset is identical to the TWiki setting in configure - that way Apache will send out the same header as TWiki would have done. Since there is a site-wide character set for TWiki currently, this should work fine.

-- RichardDonkin - 27 Apr 2008

I would like to use this addon, but I don't have the rights to use 'install' on the server which hosts my site (and don't feel like convincing the admins to do the work for me). Is there any other way of installing it - e.g., by adding the patches to the scripts oneself?

-- FabianDorsch - 27 Aug 2008

I'm having some difficulty getting this one off the ground. Installed it every way possible. In all cases, it can't run pcad?clear. Here is the result of the install:

[copswiki@rpdz2 twpc]$ ./install -q -O -v /home/copswiki/www/twiki/bin
Installing twnc
Compiling pccr.c
Warning: could not clear cache by calling http://www.copswiki.org/twiki/bin/pcad?clear
Check that the binurl= line is correctly defined in /home/copswiki/www/twiki/bin/pc-config
Install Ok. FYI admin page is at http://www.copswiki.org/twiki/bin/pcad
[copswiki@rpdz2 twpc]$ cat /home/copswiki/www/twiki/bin/pc-config

Which is correct. When I just try to run pcad from the command line, I get a blank response.

There are no hints in error logs, etc.

Trying to access pcad from the browser gives 500 error.

-- RaymondLutz - 25 Nov 2008

I changed the ModificationPolicy of this extension from ContactAuthorFirst to PleaseFeelFreeToModify due to inactivity. Anyone please feel free to work on this extension.

-- PeterThoeny - 2012-12-03

Edit | Attach | Watch | Print version | History: r61 < r60 < r59 < r58 < r57 | Backlinks | Raw View | Raw edit | More topic actions
Topic revision: r61 - 2012-12-03 - PeterThoeny
  • Learn about TWiki  
  • Download TWiki
This site is powered by the TWiki collaboration platform Powered by Perl Hosted by OICcam.com Ideas, requests, problems regarding TWiki? Send feedback. Ask community in the support forum.
Copyright © 1999-2018 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.