create new tag
, view all tags

Feature Proposal: Control over Variable Expansion


We need more control over variables that should get expanded on topic creation.

Use case:

The "My links" section of new user homepages has this:

   * %TWIKIWEB%.ATasteOfTWiki - view a short introductory presentation on TWiki for beginners
   * %TWIKIWEB%.WelcomeGuest - starting points on TWiki
   * %TWIKIWEB%.TWikiUsersGuide - complete TWiki documentation, Quick Start to Reference
   * Sandbox.%HOMETOPIC% - try out TWiki on your own
   * Sandbox.%TOPIC%Sandbox - just for me

Although accurate and correct to use TWiki variables, it can be initimidating to new users. For education purposes it is better to expand the variables (just in the "My links" section) to show:

   * TWiki.ATasteOfTWiki - view a short introductory presentation on TWiki for beginners
   * TWiki.WelcomeGuest - starting points on TWiki
   * TWiki.TWikiUsersGuide - complete TWiki documentation, Quick Start to Reference
   * Sandbox.WebHome - try out TWiki on your own
   * Sandbox.JimmyNewtronSandbox - just for me

That is, for better usability we need to have some control on which variables get expanded.

Description, Documentation, and Example

When you create a new topic that is based on a template topic, a number of TWikiVariables are expanded, such as %DATE%, %WIKIUSERNAME% or %URLPARAM{"name"}%. See complete list at TWikiTemplates#TemplateTopicsVars. It should be possible to expand any desired variable.

This is possible with a new %STARTSECTION{ type="expandvariables" }% ... %ENDSECTION{ type="expandvariables" }% section that can be put into template topics. Example:

%STARTSECTION{ type="expandvariables" }%
   * %TWIKIWEB%.ATasteOfTWiki - view a short introductory presentation on TWiki for beginners
   * %TWIKIWEB%.WelcomeGuest - starting points on TWiki
   * %TWIKIWEB%.TWikiUsersGuide - complete TWiki documentation, Quick Start to Reference
   * Sandbox.%HOMETOPIC% - try out TWiki on your own
   * Sandbox.%TOPIC%Sandbox - just for me
%ENDSECTION{ type="expandvariables" }%

This is a first step. At a later point we can add a feature to have fine-grained control over what variables get expanded, such as with %STARTSECTION{ type="expandvariables" expand="all, -URLPARAM" }%. This feature is out of scope of this proposal!

(Note: Above syntax replaces the original proposal of %EXPANDVARS{text with variables}% syntax)

Impact and Available Solutions


This is tracked in Bugs:Item3647

-- Contributors: PeterThoeny


What happens if you have some variables on a line that you want to expand and some that you don't?

-- JohnRouillard - 16 Feb 2007

With first proposal:


With second proposal:

%STARTSECTION{type="expandallvars"}% %TWIKIWEB% %ENDSECTION{type="expandvars"}%, left alone %TOPIC%, %STARTSECTION{type="expandallvars"}% %WEB% %ENDSECTION{type="expandvars"}%

-- PeterThoeny - 16 Feb 2007

This brings up interesting use cases:

  • Create topics that archive web site content by expanding an INCLUDE
  • Create a user manual by taking a snapshots of TWiki content with TOC and multiple INCLUDEs
  • Create topics that take snapshots of TWiki content by expanding a SEARCH

-- PeterThoeny - 16 Feb 2007

And how to code the next line with second proposal, keeping in the mind do not expand %TOPIC%?

%EXPANDVARS{%MAKETEXT{This is a [_1] topic." args="%TOPIC%"}%}% 

It would probably be nice to write

%STARTSECTION{type="expandallvars"}%%MAKETEXT{This is a [_1] topic." args="%<nop>TOPIC%"}%%ENDSECTION{type="expandvars"}%
to get
This is a %TOPIC% topic.
and to write
%STARTSECTION{type="expandallvars"}%%MAKETEXT{This is a [_1] topic." args="%<nop><nop>TOPIC%"}%%ENDSECTION{type="expandvars"}%
to get
This is a %<nop>TOPIC% topic.

It would certainly be nice to have some fixed, clear, flexible and documented common way to manage variables expansion order.

-- SergejZnamenskij - 16 Feb 2007

Yes, that is the idea. Actually, the documented way is to escape variables with a %NOP% which gets removed at time of instantiation.

-- PeterThoeny - 17 Feb 2007

Embedding a variable in another conditional expansion context is an extremely bad idea. This is simple to fix, by writing

instead of

While I like the usability of Peter's proposed expandvars section, I have some concerns that from a performance perspective, sections should be discouraged as they require multi-pass processing of the topic text, which is an expensive process.

Having said that, I like the section approach best. Thinking a bit about the implementation, at the moment we have two functions that do variable expansion; expandTagOnTopicRendering and expandTagOnTopicCreation. The first handles tag expansion during normal rendering, the second during template expansion. The only difference between the two is that the template expansion version has a list of the variables it is allowed to expand. This suggests what might be a more general approach to this problem, viz:

This would temporarily add the variable "SYSTEMWEB" to the list of variables expanded on section expansion, and importantly, remove URLPARAM.

Stepping beyond this, the concept of conditional expansion sections is attractive for INCLUDE as well. For example, in the case where I want ControlOverVariableExpansion in an included topic to be expanded in the context of the parent topic instead of the included topic, I might write:

to suppress expansion when the topic is included (i.e. delay it until the final page rendering). Consistent with the view that INCLUDE is really just a way of creating sections, you should also be able to write this:
%INCLUDE{"AnotherTopic" expand="-TOPIC"}%
Thinking even further, this gives us a fresh insight into the nature of template topics. They are actually just ordinary topics that happen to be included at the time a topic is being created. If we think about them this way we have the opportunity to create a clean and consistent syntax and use-model.

-- CrawfordCurrie - 17 Feb 2007

Nice thoughts, Crawford. I like the proposed syntax and the reasoning behind it: they are both very clear and sensible. There have also been questions recently on how to get ControlOverVariableExpansion in EditTablePluginDev. It would be nice if the same logic that is built here would also extend and apply to new rows inserted by the EditTablePlugin.

-- KeithHelfrich - 17 Feb 2007

Crawford, you are entitled to your opinion. The idea of the "extremely bad idea" is to be able to write
%EXPANDVARS{Simply a complete sentence in the %TOPIC% that shows %SEARCH{....}% located in the %WEB% web.}%
That is basically how the %CALC{}% of the SpreadSheetPlugin can be used, and it has shown to be very useful (e.g. you do not need to repeat a CALC for every little fomula.)

I like the idea of the %STARTSECTION{ expand="+SYSTEMWEB, -URLPARAM" }% syntax, although it is programmers centric, and limits use cases unless we expand the syntax. For example, I might want to create an app that creates snapshots of some TWiki content. As the app writer I cannot predict what variables the users are going to use.

And besides, how would you make the distinction of normal topic view vs template topic instantiation? The whole point of this topic is to have control over the variable expansion at template topic instantiation.

-- PeterThoeny - 17 Feb 2007

Sorry, I should have made it clear that my point isn't an opinion, it's a fact. The variable expansion parser operates inside-out-left-right, which means that a %TWIKIWEB% embedded in %EXPANDVARS{%TWIKIWEB%}% will be expanded before the enclosing EXPANDVARS. We encountered this situation before with %IF, and it's a known limitation. Yes, I agree that it would be very useful if it didn't always work this way; it would allow for a nicer syntax in this case, and %IF could leverage it as well. However I am pointing out that it does work this way, and would require a significant amount of work to change. No need to jump down my throat.

There are three cases of variable expansion I believe you need to cater for:

  1. template instantiation
  2. include
  3. normal topic rendering
There may be others, I'm not sure. There are a number of ways of detecting these conditions in code. The method most often used is detection from the caller context. e.g. TWiki::expandTagsOnTopicCreation versus TWiki::expandTagsOnTopicRendering, similar for expansion on INCLUDE, communicated to called functions using context_identifiers. For example:
sub instantiateTopicTemplate {

sub includeTopic {

sub expandTags { ... if (inContext('template_topic_instantiation')) { ... expand the vars normally expanded in templates plus the SECTION expand= changes ... } elsif (inContext('including_subtopic')) { ... expand the vars normally expanded in includes plus the INCLUDE expand= changes ... } else { ... expand the vars normally expanded (plus any relevant SECTION expand= changes?) ... } } Of course it's a lot cleaner to pass in a handler object instead of switching on a context, but the architecture of the code makes that difficult.

From a user perspective, do you need to support different behaviour in the dfferent contexts? If you do, then you could use an %IF:

%STARTSECTION{ %IF{"context template_topic_instantiation" then="expand=\"+SYSTEMWEB, -URLPARAM\"}% }%

or you could make it explicit in the section parameters (not sure what the ramifications of this would be)

%STARTSECTION{ if="context template_topic_instantiation" expand="+SYSTEMWEB, -URLPARAM" }%

-- CrawfordCurrie - 18 Feb 2007

Actually, as a matter of fact, if you write in a template topic this:
This is a complete sentence in the %TOPIC% that shows %SEARCH{....}% located in the %WEB% web.
then a topic instantiation gets exactly that sentence with unexpanded variables, e.g. variables %TOPIC%, %SEARCH{....}% and %WEB% do not get expanded (by design). Hence the proposal to pack this sentence in %EXPANDVARS{...} (proposal one), or %STARTSECTION{type="expandallvars"}% ... %ENDSECTION{type="expandallvars"}% (proposal two) so that they get expanded. In normal topic view, %EXPANDVARS{...} (or the other syntax) does a "no operation", hence no performance impact.

Whatever syntax we come up with, it should be:

  • easy to remember for users
  • easy to understand (KISS preferred over many bells and whistles)
  • address the actual need (control over variables during template expansion)
  • support expansion of all variables (app writer cannot predict what variables the user is going to use)

-- PeterThoeny - 18 Feb 2007

And what about

where the first ALL without sign is a reserved word?
This is a complete sentence in the %TOPIC% that shows %SEARCH{....}% located in the %WEB% web. 

-- SergejZnamenskij - 19 Feb 2007

Good idea. * is a good convention for ALL, as you don't need reserved words then.

%STARTSECTION{ ... expand="*, -URLPARAM"}%

-- CrawfordCurrie - 20 Feb 2007

A star is not so good because it could be confused with a wildcard, such as expand="TOPIC*". Other vars use already keywords, such as web="all", limit="all" or reverse="on" of SEARCH. So, it is more consistent to write expand="all, -URLPARAM".

-- PeterThoeny - 25 Mar 2007

We need to come to an agreement on the spec OR put forward a number of clear proposals we can vote for to take this one to the release meeting. Peter seems to be a clear driver on this one.

-- KennethLavrsen - 08 Apr 2007

I refined the spec above. I think it addresses all points, including the "very bad idea" voice.

-- PeterThoeny - 09 Apr 2007

OK let us take it to release meeting

-- KennethLavrsen - 23 Apr 2007

Accepted at release meeting after intense discussion if the feature should be plugin.

I would like to add a personal note. I find it strange that a proposal exists for 2 months and the spec discussed and agreed, and then at a release meeting it is suddenly proposed to make the feature a plugin. If we talk a default plugin which is always (in practical) enabled then I do not at all understand why. That would be a bad technical implementation of such a small feature enhancement of something already in core with a very negative performance impact (plugin init cost). And if we talk non-default plugin then it is in reality a rejection.

Please try and voice this type of counter proposals in the proposal topic.

It ended up that only two votes and it became a YES. FreetownReleaseMeeting2007x04x23

-- KennethLavrsen - 23 Apr 2007

Personal note: A concur with Kenneth. It was kind of strange to get yet another concerns raised, this after 2.5 month of discussion! The least to say, it does not make me feel motivated if I get this much resistence for each and every well documented feature I propose. I already changed the spec twice based on the concerns raised. The new process works in general, but only if people give timly feedback. This time it felt more like yet one more roadblock, for the n-th time.

-- PeterThoeny - 24 Apr 2007

Code changes are done. Doc changes and homepage change is pending.

-- PeterThoeny - 28 May 2007

Please consider Bugs:Item1855 when you do this. Thanks.

Where are we with this? I'm not sure what the spec is, I don't see any code in the repository, and I suspect it's going to be tough to test.....

-- CrawfordCurrie - 22 Jun 2007

Peter never finished this implementation. There is some code in MAIN branch but not in TWikiRelease04x02 branch.

As I write this we are 7 months into a code freeze so this is obviously deferred to GeorgetownRelease.

-- KennethLavrsen - 17 Dec 2007

Yep, this implementation seem to have stalled. I have only limited time, too much time went into justifying and refining the spec. Bummer it did not make it into TWiki 4.2 (see Support.ExpandVariablesOnTopicCreation). I hope someone has time to finish this.

-- PeterThoeny - 23 May 2008

It was only Doc changes and homepage changes that was missing as I recall it. You should finish that yourself Peter - now in trunk. You cannot expect someone else to document the code you added now exactly one year ago.

-- KennethLavrsen - 23 May 2008

Good point, yet another sleepless night...

-- PeterThoeny - 23 May 2008

Actually, it looks like this feature is in 4.2, but it is not documented. Above doc with %STARTSECTION{ type="expandvariables" }% ... %ENDSECTION{ type="expandvariables" }% is accurate.

-- PeterThoeny - 20 Dec 2008


  • TWiki.TWikiTemplates
  • TWiki.NewUserTemplate from TWikiUserMappingContrib

-- SopanShewale - 30 Jan 2009

Edit | Attach | Watch | Print version | History: r29 < r28 < r27 < r26 < r25 | Backlinks | Raw View | Raw edit | More topic actions
Topic revision: r29 - 2009-02-16 - PeterThoeny
  • Learn about TWiki  
  • Download TWiki
This site is powered by the TWiki collaboration platform Powered by Perl Hosted by OICcam.com Ideas, requests, problems regarding TWiki? Send feedback. Ask community in the support forum.
Copyright © 1999-2018 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.