caching1Add my vote for this tag create new tag
, view all tags

Patch Proposal: Allow browser to properly cache viewed pages


Caching is often a good idea, and caching on the client browser has no inconsistencies due to user-specific customization. But, not all TWiki content should be cached (if it uses dynamic SEARCH, GMTIME, etc., or if it includes topics that are dynamic).


Browsers will cache pages that don't use a POST or a query string, if the webserver returns a proper Last-Modified: header. This patch includes a smart LastModifiedPlugin that required various changes to the lib/*.pm. The API for building additional headers through a plugin simply didn't work. I also modified handleCommonTags to return a flag indicating if any dynamic tags were processed. The plugin gives a (configurable) expiry time of 0 seconds for dynamic pages (and handles includes properly). It's also careful not to give expiry times in the future for scripts like edit wink

After enabling caching, some bugs were found in the behavior of the edit/preview/save trio (even though PUSH was used, things were mysteriously getting cached). The method of using ?t=3294987 time nonces was extended with javascript to add in the client side time, which helps when one of the view pages generating the edit link is cached. I did not break anything for non-javascript browsers (using the NOSCRIPT tag). The nonces were added to the preview and save links, just for good measure (even though they should not be necessary, they were needed on some browsers).


Two new preferences were introduced:

  • Set EXPIRESINSEC = 600

The first is the number of seconds static content should be cached; the second is for content that either directly or indirectly (through includes) uses any tags or variables that change from request to request.

These variables can be set on a per-page basis (much like ALLOWTOPICCHANGE) and will override web/global settings.

You may be confused if you modify a page and come back to it later by clicking on a WikiWord link to it within EXPIRESINSEC seconds ... you will see an older, cached version (clicking save will show you the most recent version, and the browser history will work as expected, though). A browser refresh should always give you the most recent version (shift- or control- refresh might be needed if you are working through a caching proxy, though).

In your edit.tmpl and preview.tmpl, use %FORMPREVIEWTOPIC% and %FORMSAVETOPIC% respectively, instead of

<form name="main" action="%SCRIPTURLPATH%/save%SCRIPTSUFFIX%/%WEB%/%TOPIC%" method="post">
. (and of course use %EDITTOPIC% for the edit link in view.tmpl)


Created lib/TWiki/Plugins/LastModifiedPlugin.pm which inserts a proper Last-Modified header (along with an ETag to indicate that it is a weak tag; that is, the byte content is not guaranteed to be identical for the same Last-Modified although the semantic content is), and, depending on the configuration variables described above, inserts an Expires: header depending on whether the content is static or dynamic.

Created lib/WebDebug.pm (usage: WebDebug::debug($a,$b,"blah")) which not only writes output to data/debug.txt, but displays it inline in the web browser when SCRIPT_NAME contains "debug" (or at the command line if you are debugging scripts from there). I created a "/twiki/bin-debug" alias on my webserver to do this. In order to display output that comes before the header is output, WebDebug::webHeadersDone must be called after your header is printed (actually, no output will appear in the browser until it is).

Added &getPrefFromText to lib/TWiki/Func.pm (which efficiently recognizes variables like EXPIRESINSEC in topic text)

Modified &TWiki::Plugins::commonTagsHandler( $text, $theTopic, $theWeb, $wasInclude, $dynamic ); so that the last argument is 1 if the text included any dynamic tags (the fourth was already set, indicating whether it was called from an include or the main topic, but wasn't documented). Modified the onlyOnce plugins to properly return values other than scalars (for instance, LastModifiedPlugin returns a hash/list of additional headers)

In TWiki.pm, fixed the way web headers were built (it was pretty bad in there) and used the result of the plugin call. Added alt_javascript_text($pretext,$alttext,$posttext,$javascript ) and alt_javascript_nonce to essentially allow NOSCRIPT/SCRIPT alternate generated text inside of tags e.g. FORM, A HREF ...

And of course, Save.pm and Preview.pm (along with new templates for them) to implement the new nonces that avoiding unwanted caching seemed to require.

Note: I've observed several instances where the store topic implementation for rcswrap silently fails (this was definitively not a caching issue) but have been unable to reproduce reliably. I've been using RcsLite since and haven't run into it again. I suspect something about file or rcs lock ownership as the root cause. Silent failures are bad wink

Note: Patch is attached as http://twiki.org/p/pub/Codev/BrowserCacheFixes/graehl-twiki-browser-cache.diff. The patch is against the TWikiBetaRelease of 07 May 2004.

-- JonathanGraehl - 13 Jul 2004


Thanks for sharing this. We have not looked into details, also on performance. These are extensive changes on the core code, which needs to be tested in different environments (also non-cgi). The CoreTeam is now working on the CairoRelease, there is not enough time to look into this patch before the release.

-- PeterThoeny - 14 Jul 2004

Jon, FWIW I think this is really useful work, given the end-user performance of TWiki at the moment. SvenDowideit want Dakaar to be a "performance release" so this definitely fits there, so I'm going to unilaterally assign it to that release. A core team member can always unassign it again! wink

-- CrawfordCurrie - 14 Jul 2004

Thanks. I think it's a good decision to release Cairo ASAP wink I hope that the core parts of this can be merged into Dakaar before extensive refactoring makes merging more painful (I consider the core of this patch to be detecting dynamic tags and passing $dynamic_or_static as an argument to the plugin commonTagsHandler, and the javascript+server-side nonce edit/preview/save URL generation; the LastModifiedPlugin has no other dependencies except WebDebug (which just requires an init from TWiki.pm in a mod_perl environment) and getPrefText, which I thought would be a useful addition to Func.pm but could certainly be put inside LastModifiedPlugin instead.

Given the heft (in terms of number of modules and lines of code) of the TWiki codebase, I hope Dakaar will focus on refactoring to the extent that the initialization of global variables is cleaned up enough to have confidence that all the scripts will run under mod_perl or similar perl-image-reusers. I suppose an alternative mod_perl type speedup could be devised that doesn't reuse perl images, but simply keeps a cached of preforked/preloaded ones (this wouldn't help throughput under sustained load, but would have the same benefit for latency).

-- JonathanGraehl - 14 Jul 2004

This is similar to CacheControlHeaders and BrowserAndProxyCacheControl.

-- RichardDonkin - 15 Feb 2006

TopicClassification FeatureRequest
TopicSummary Improve browser page cacheing
CurrentState UnderInvestigation
OutstandingIssues There is no point in assigning this to a release until it has a CoreTeam priority and someone is working on it.
RelatedTopics CategoryCaching



Topic attachments
I Attachment History Action Size Date Who Comment
Unknown file formatdiff graehl-twiki-browser-cache.diff r3 r2 r1 manage 38.7 K 2004-07-14 - 02:11 JonathanGraehl the diff/patch
Edit | Attach | Watch | Print version | History: r9 < r8 < r7 < r6 < r5 | Backlinks | Raw View | Raw edit | More topic actions
Topic revision: r9 - 2006-02-15 - RichardDonkin
  • Learn about TWiki  
  • Download TWiki
This site is powered by the TWiki collaboration platform Powered by Perl Hosted by OICcam.com Ideas, requests, problems regarding TWiki? Send feedback. Ask community in the support forum.
Copyright © 1999-2018 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.