Question

I had a simple requirement that was extremely difficult to implement in TWiki. It's the kind of requirement that I believe is common and should be easier. I'm filing this as a support bug for now, and once I know the answer I'll probably want to create a documentation bug so I can publish the answer to other users.

First, here are my specific requirements:

1. I want a form that has the fields FirstName, LastName, Address, Latitude and Longitude.
2. I want to display an address book that includes a link to Google maps. The link should only be shown if the form contains either a coordinate location or an address location. The coordinates fields should be used to find the location, and if the coordinates are blank then the fall-back should be the Address field.
3. I want for the address book to work no matter what characters people enter into their address form. Note that this is the hard requirement!

My first two requirements may be specific to my application, but the third is pretty universal: data entered on one form should not break the search results for everybody.

Next, let me show the solution that I finally came up with for these requirements: (The original code doesn't contain any line breaks but I've added them for readability.)

%SEARCH{"%META:FORM.*[C]ontactInfo" nonoise="on" regex="on"
header="| *Name* | *Address* |"
format="| <noautolink>$formfield(FirstName) $formfield(LastName)</noautolink> |
<!-- Check for a location, whether it is an address or coordinates.
Do a URL-encoding to remove newlines and an HTML-encoding to remove single quotes. -->
$percntIF{\"'$percntENCODE{\"$percntENCODE{\"$formfield(Address)\" type=\"url\"}$percnt\" type=\"entity\"}$percnt'!='' or
'$percntENCODE{\"$percntENCODE{\"$formfield(Longitude)\" type=\"url\"}$percnt\" type=\"entity\"}$percnt'!='' and
'$percntENCODE{\"$percntENCODE{\"$formfield(Latitude)\" type=\"url\"}$percnt\" type=\"entity\"}$percnt'!=''\"
then=\"<!-- There is a location. Coordinates are preferred if they are present. -->
[[http://maps.google.com/maps?f=q&hl=en&q=
$percntIF{\"'$percntENCODE{\"$percntENCODE{\"$formfield(Longitude)\" type=\"url\"}$percnt\" type=\"entity\"}$percnt'!='' and
'$percntENCODE{\"$percntENCODE{\"$formfield(Latitude)\" type=\"url\"}$percnt\" type=\"entity\"}$percnt'!=''\"
then=\"$percntENCODE{\"$formfield(Latitude), $formfield(Longitude)\" type=\"url\"}$percnt\"
else=\"$percntCALC{\"$dollarSUBSTITUTE($percntENCODE{\"$formfield(Address)\" type=\"url\"}$percnt,
$percntENCODE{\"$percntBR$percnt\" type=\"url\"}$percnt,
$percntENCODE{\" \" type=\"url\"}$percnt)\"}$percnt\"}$percnt
$percntENCODE{\" ($formfield(FirstName) $formfield(LastName))\" type=\"url\"}$percnt
&z=15&ie=UTF8&om=1][map]]\"
else=\"<!-- There is no location. -->\"}$percnt
|" newline=" "}%

You might notice that this is a large amount of code for such modest requirements. I believe that the problem lies in the complex expansion rules for variables. As you can imagine, I started with a simple solution and then had to start adding work-arounds for various problems:

1. I had to add <noautolink> around the name field because some last names looked like WikiWords.
2. Inside the condition of the IF, I had to URL-encode the latitude and longitude form fields. Otherwise newlines in these fields would break the IF syntax.
3. Also inside the condition of the IF, I had to HTML-encode the latitude and longitude form fields. I did this after URL-encoding them. Otherwise single quotes in these fields would break the IF syntax.
4. Inside the "then" parameter of the IF, I had to substitute %BR% with a space. Otherwise, newlines in the address field might show up as <br /> in the link. This might be a bug where the "newline" parameter to IF is not working. It actually works in most cases, but fails in the case where a form field contains a variable that expands to a multi-line string.
5. Before trying to do a substitution I had to URL-encode the address and the %BR% and the space. If these parameters were not all URL-encoded then a newline in the address field would break the SUBSTITUTE syntax.

I think that covers it. Now, let me summarize by saying why I am not happy with this behavior:

1. I think that it is a common requirement for search results to work regardless of the values of the form fields. We should make this easy for admins to accomplish. If this remains too difficult for admins then broken TWiki pages will be too common a sight. I'd love to hear suggestions on this.
2. I didn't know the rules for when certain variables get expanded. For example, will variables inside a SUBSTITUTE formula get expanded before or after the substitution? If a form field contains a newline, then will that get replaced with <br /> before or after the SEARCH formatting is done? I can't find answers in the documentation. It would be great if someone could answer this question and then the documentation could be improved.

Environment

-- ChadParry - 13 Dec 2006

Answer

If you answer a question - or someone answered one of your questions - please remember to edit the page and set the status to answered. The status selector is below the edit box.

You are absolutely right, Chad. It can be a complete PITA to do this kind of thing. Did you investigate the potential for using the SpreadSheetPlugin to help out? It has a huge number of obscure but very clever functions.

BTW the rules for expansion are actually very simple; inside-out-left-right. i.e. if you have a %TAG{ ... ]% with an %INNERTAG{ ... }% then all recognisable tags in the ... will be expanded before the %TAG itself is expanded. Tags on the left are expanded before tags on the right.

-- CrawfordCurrie - 14 Dec 2006

Thanks, the inside-out rule will help. Here's a related issue. A tricky thing is figuring out whether a tag inside quotes will be expanded before the quotes are parsed (meaning that quotes inside the expansion will create an error) or after the quoted string is identified (meaning quotes inside the expansion are OK). For example:

   * Set INPUT = Single' and double" quotes in a string
   * Set THISWORKS = %IF{"'%INPUT%'=''" then=""}%
   * Set THISBREAKS = %IF{"''=''" then="%INPUT%"}%

Also, the spreadsheet functions were usually not an option because they break on any strings that contain a newline. I haven't found a good solution yet. Potential ways to fix this are (in order of importance):

• Identify quote delimiters for parameters before expanding variables.
• Allow the IF conditional to contain newlines.
• Make the ENCODE function also encode single quotes in URL strings.
• Enforce better consistency on when a newline gets replaced with %BR%.
• Allow the spreadsheet expressions to span multiple lines and contain any punctuation.

-- ChadParry - 21 Dec 2006

Chad, first of all, thanks for the detailed feedback. It must have taken you ages to come up with this solution. We are trying to make it easier for WikiChampions to create TWikiApplications.

On your suggestion to enhance the SpreadSheetPlugin:

• Supporting quote delimters: This might be challenging to implement with a regex parser. The current regex based parser is very fast, changing that to a token parser will slow down the plugin ten-fold.
• Newline support: This can be done, after carefully checking that it does not break existing content.

-- PeterThoeny - 24 Dec 2006

Chad, please feel free to enhance the TWiki documentation (distribution doc and SupplementalDocuments)

-- PeterThoeny - 02 Jan 2007

I know this was last updated a while ago, but I have had a similar issue with the SpreadSheetPlugin choking on newlines. The solution was to add the newline="..." parameter to the SEARCH function. This allowed me to deal with values that contained newlines and carriage returns w/o problems.

Chad, if you see this, I hope it helps

-- MatthewCardozo - 24 Oct 2008