Search Pattern Cookbook

The Search function in TWiki is very powerful. Especially searches using a RegularExpression play an important part of tapping TWiki's full potential. Unfortunately RegularExpressions can be incredibly obscure to the uninitiated.

Most people not familiar (enough) with Regular Expressions mostly cut and paste (and maybe tweak) from existing examples. This page intends to collect lots of examples together.

Pattern 1: Extract values from a table

Problem definition

Suppose there is a topic with a table defining entries in a TWikiForm. I.e. they define select menu items in a form template. They are then formatted like:

| *Name* | *Type* | *Tooltip message* |
| option1 | option | |
| option2 | option | |
| option3 | option | |

How to extract the 'name' values, i.e. 'option1', 'option2' and 'option3' and put them in a HTML form select input?

Solution

The following search pattern can be employed:

<form>
<select>
%SEARCH{ "^\|[^\|]*\| *option *\|" topic="%TOPIC%" type="regex" multiple="on" nosearch="on" nototal="on" format="<option>$pattern(^\| *(.*?) *\|.*)</option>" }%
</select>
</form>

which is, in effect:

option1 option2 option3

Pattern 2: List generated from form classification

Problem

Imagine a TWiki form-based topic classification, i.e. every page has a form with several fields. How to:

1. create a search to display all topics where one form field is set to a certain value
2. create a search to filter the list above based on the values of a second form field

Test case

In practice:
Image a TWiki form with two fields:

• TopicClassification = One, Two or Three
• TopicStatus = Test or Final

We will:

1. List all topics where the TopicClassification field is set to 'Two'
2. Enable the user to filter this list based on the values of TopicStatus

Solution

%SEARCH{"[T]opicClassification.*value\=.*Two;[T]opicStatus.*value\=.*%URLPARAM{type}%"
type="regex" casesensitive="on" nosearch="on" 
format="   * $topic - <font face=\"arial,helvetica\" size=\"1\"> 
_last modified by_ $wikiusername _on_ $date </font> %BR% &nbsp;&nbsp;&nbsp; 
<font face=\"arial,helvetica\" size=\"1\"> $formfield(TopicStatus) </font>" 
sort="topic"}%

The filtering select dialogue is created as in Pattern 1:

%STARTSIDEBAR%
*Filter:* %BR%
<form name="selectType" action="%SCRIPTURLPATH{"view"}%/%WEB%/" >
<select name="type" size="1" onchange="document.location=this.value;"> 
%SEARCH{ "^\|[^\|]*\| *option *\|" topic="TopicClassification" web="%WEB%" type="regex" 
multiple="on" nosearch="on" nototal="on" format="<option value=%INCLUDINGTOPIC%?type=$pattern(^\| *(.*?) *\|.*)>$pattern(^\| *(.*?) *\|.*)</option>" }% 
<option value=%INCLUDINGTOPIC%>All pages</option> </select>
</form>
%STOPSIDEBAR% 

This will create similar functionality as TWiki:Plugins.TopicClassificationAddOn

Pattern 3: Creating lists of TWiki usernames

Problem

How to populate a list box with all usernames of registered TWiki users

Solution 1: Appropriate for Sep 2004 TWiki (Cairo)

<form name="testing" action="%SCRIPTURLPATH{"view"}%/%USERSWEB%" method="get">
<select name="topic">
<option>Select user...</option>
%SEARCH{ "Name:;Email:;Country:" web="%USERSWEB%" type="regex" nosearch="on" nototal="on" format="<option>$topic</option>" }%
</select>
<input type="submit" value="Go" />
</form>

Which expands to this: (here limited to all Z* users because TWiki.org has so many)

Select user... ZHLeoLi ZacAlfonsoBrady ZacGowen ZachCrawford ZachFrey ZachGarner ZachLoch ZacharyCrowell ZacharyHamm ZacharyKoziol ZackBronson ZackBrown ZahTally ZaidAlBanna ZainMagsi ZainalArifin ZainalHakim ZakFerrari ZakWilliamson ZakariaHaris ZakiAlam ZalTarang ZalemMelaz ZandraHaes ZaneleMotsa ZariusTularial ZbigniewChamski ZbigniewKosowski ZbigniewKulesza ZbigniewLukasiak ZbynekCerveny ZdenekPalecek ZdenkoVarnus ZebCook ZedTiberius ZeebaBabu ZeevBelkin ZekengLiang ZelalemMengistu ZeljkoBlace ZeljkoStevanovic ZemianDeng ZenderChen ZhanYi ZhangBin ZhangJun ZhangKai ZhangQiang ZhangSeven ZhangZ ZhangZhanj ZhannaNakhshunova ZhaoHuijuan ZhenLiu ZhengLingxiang ZhengShao ZhiWang ZhiWang63 ZhihengZ ZhikunLiu ZhoanMoro ZhongZputee ZhonglinKo ZhouLi ZhouYiZheng ZhouYuanchi ZhuChenyang ZhuTao ZhuXiaoming ZiaMulla ZiaQuraishi ZiaSyed ZiadMelhem ZianWang ZibiJanik ZierohTardy ZigmokBenevolence ZihongLu ZisuZhao ZivBaram ZiyinZhang ZlatinIvanov ZoeBCullen ZoeCullen ZoharGolan ZoharMelamed ZolarLee ZoltanGabor ZoltanSzucs ZoltanZidarics ZoranKajis ZoranPavlovic ZoranStojanovic ZsoltLipjanszky ZsoltMolnar ZsoltSzloboda ZubairKhan ZubairSaiyed ZviBoshernitzan ZviZemel ZvikaRap ZyadShukoor

This searches all topics in the Main web that contain "Name", "Email" and "Country" bullets. Alternatively, do a FormattedSearch with multiple="on" on the Main.TWikiUsers topic.

Solution 2: As Solution 1, but with possibility for multi-selecting usernames

The example of Solution 1 produces the list box. Add a MULTIPLE to the select statement, i.e.:

<select name="topic" size="2" MULTIPLE>

Please note that the Search pattern is unchanged compared to Solution 1. The change is in the HTML form element.

The abovementioned modification is, in effect:

Select user... ZHLeoLi ZacAlfonsoBrady ZacGowen ZachCrawford ZachFrey ZachGarner ZachLoch ZacharyCrowell ZacharyHamm ZacharyKoziol ZackBronson ZackBrown ZahTally ZaidAlBanna ZainMagsi ZainalArifin ZainalHakim ZakFerrari ZakWilliamson ZakariaHaris ZakiAlam ZalTarang ZalemMelaz ZandraHaes ZaneleMotsa ZariusTularial ZbigniewChamski ZbigniewKosowski ZbigniewKulesza ZbigniewLukasiak ZbynekCerveny ZdenekPalecek ZdenkoVarnus ZebCook ZedTiberius ZeebaBabu ZeevBelkin ZekengLiang ZelalemMengistu ZeljkoBlace ZeljkoStevanovic ZemianDeng ZenderChen ZhanYi ZhangBin ZhangJun ZhangKai ZhangQiang ZhangSeven ZhangZ ZhangZhanj ZhannaNakhshunova ZhaoHuijuan ZhenLiu ZhengLingxiang ZhengShao ZhiWang ZhiWang63 ZhihengZ ZhikunLiu ZhoanMoro ZhongZputee ZhonglinKo ZhouLi ZhouYiZheng ZhouYuanchi ZhuChenyang ZhuTao ZhuXiaoming ZiaMulla ZiaQuraishi ZiaSyed ZiadMelhem ZianWang ZibiJanik ZierohTardy ZigmokBenevolence ZihongLu ZisuZhao ZivBaram ZiyinZhang ZlatinIvanov ZoeBCullen ZoeCullen ZoharGolan ZoharMelamed ZolarLee ZoltanGabor ZoltanSzucs ZoltanZidarics ZoranKajis ZoranPavlovic ZoranStojanovic ZsoltLipjanszky ZsoltMolnar ZsoltSzloboda ZubairKhan ZubairSaiyed ZviBoshernitzan ZviZemel ZvikaRap ZyadShukoor

Solution 3: Appropriate for TWiki 4 (Dakar)

When the User information is stored in a UserForm (as is default in Dakar) then this list can be generated as follows:

<form name="testing" action="%SCRIPTURLPATH{"view"}%/%USERSWEB%" method="get">
<select name="topic">
<option>Select user...</option>
%SEARCH{"%META:FORM.*[U]serForm" web="%USERSWEB%" type="regex" casesensitive="on" nosearch="on" format="<option>$topic</option>" sort="topic" excludetopic="Test*, TWiki*"}%
</select>
<input type="submit" value="Go" />
</form>

In the above example:

• META:FORM.*[U]serForm will search for all topics with a UserForm attached - change this if you have a different form where userdata is stored. Please note that this search does not actually extract anything from the form - it just uses it to identify the appropriate pages
• excludetopic="Test*, TWiki*" allows to skip all topics starting with Test and TWiki, such as TestUser or TWikiAdmin. Use this if you have any special users who you do not want appearing in this list

Pattern 4: Extract the parent of a given topic

Problem

How to get to the parent of the current topic to display on the page?

Solution 1: Using META

Since TWiki 4.0 you can now use the META variable:

%META{ "parent" dontrecurse="on" }%

Solution 2: Using SpreadSheetPlugin

You might think that the following Search would do the trick:

%SEARCH{ "^%BASETOPIC%$" scope="topic" nosearch="on" type="regex" nonoise="on" format="   * $parent" }%

However, the $parent link fails if the topic has no parent set ($parent will be empty). You can use some TWiki:Plugins/SpreadSheetPlugin magic to conditionally link to the parent or to WebHome:

$percntCALCULATE{$IF($EXACT($parent,), <nop>, $NOP(   * $parent))}$percnt

So the total Search query to find a topic's parent topic is:

%SEARCH{ "^%BASETOPIC%$" scope="topic" type="regex" nonoise="on" format="$percntCALCULATE{$IF($EXACT($parent,), <nop>, $NOP(   * $parent))}$percnt" }%

Test Case

The parent topic of this topic is:

• FormattedSearch

Solution 3: Using IF statement

This pattern can be rewritten using %IF%, removing the dependency on SpreadSheetPlugin:

%SEARCH{ "^%BASETOPIC%$" web="%BASEWEB%" scope="topic" type="regex" nonoise="on" format="$percntIF{$quot$parent$quot then=$quot   * $parent$quot else=$quot<nop>$quot}$percnt" }%

Test Case

The parent topic of this topic is:

Pattern 5: Show all Children of a given topic

Problem

How to get to the list of all children of the current topic to display on the page?

Solution

The parent information is stored in the META:TOPICPARENT meta data. Do a SEARCH to find all topic parent meta data pointing to the current topic:

Children:
%SEARCH{ "META\:TOPICPARENT.*\"%TOPIC%\"" type="regex" nonoise="on" format="[[$topic]]" separator=", " }%

Note: Replace %TOPIC% with %BASETOPIC% if you put this SEARCH into the skin or a sidebar.

Pattern 6: Search and display the home topics of public webs in a list

Problem

How to find and display public webs in a drop down list box.

Solution

Thanks to TWiki:Main.PeterThoeny for these solutions.

<form>
<select name="topic">
<option value="%TOPIC%">Select...</option>
%SEARCH{ "%HOMETOPIC%" scope="topic" web="all" topic="%HOMETOPIC%" format="<option value=\"$web.$topic\">$web</option>" separator=" " }%
</select>
<input type="submit"  value="Go" />
</form>

Test case

Public webs of TWiki.

Select... Search: WebHome

Blog

Number of topics: 1

Codev

Number of topics: 1

Main

Number of topics: 1

Plugins

Number of topics: 1

Sandbox

Number of topics: 1

Support

Number of topics: 1

TWiki

Number of topics: 1

TWiki01

Number of topics: 1

TWiki02

Number of topics: 1

TWiki03

Number of topics: 1

TWiki04

Number of topics: 1

TWiki04x01

Number of topics: 1

TWiki04x02

Number of topics: 1

TWiki04x03

Number of topics: 1

TWiki05x00

Number of topics: 1

TWiki05x01

Number of topics: 1

TWiki06x00

Number of topics: 1

TWiki06x01

Number of topics: 1

WikiWed

Number of topics: 1

For private webs, or any other webs you wish to exclude from the display, use "on" for the Exclude web from a web="all" search setting in the relevant web's WebPreferences topic.

Alternative solution

This result can also be accomplished with the %WEBLIST% variable.

Select... Blog Codev Main Plugins Sandbox Support TWiki TWiki01 TWiki02 TWiki03 TWiki04 TWiki04x01 TWiki04x02 TWiki04x03 TWiki05x00 TWiki05x01 TWiki06x00 TWiki06x01 WikiWed

Pattern 7: Create a select box with values from a bullet list

Problem

We have a topic with a bullet list with category names. In another topic we want to offer these values in a select box dropdown.

For example, CategoryList has:

• Clients
• People
• Rooms
• Buildings

Solution

The following search pattern can be employed:

<select name="type">
<option>Select category...</option>
%SEARCH{"   *\s*.*?" topic="CategoryList" type="regex" multiple="on" casesensitive="on" nosummary="on" nosearch="on" noheader="on" nototal="on" format="<option>$pattern(.*   \*\s*([^\n]*).*)</option>"}%
</select>

To render the bullet list as a comma-separated list, use the separator parameter:

%SEARCH{"   *\s*.*?" topic="CategoryList" type="regex" multiple="on" casesensitive="on" nosummary="on" nosearch="on" noheader="on" nototal="on" separator="," format="$pattern(.*   \*\s*([^\n]*).*)"}%

Pattern 8: Extract a value from a named bullet list item

Problem

Display the user name in the user's topic title

Solution

Search for the Name: entry.

%SEARCH{"   * [N]ame: " topic="%TOPIC%" type="regex" casesensitive="on" nosummary="on" nosearch="on" noheader="on" nototal="on" format="---+!! $pattern(.*   \* Name: ([^\n]*).*)"}%

Test case

To create a test case, we will put a name entry here:

• Name: John Doe

Search result:

Pattern 9: Search for Form and Meta data: explained

Problem

Below is an example of a search that searches form data. The questions are:

• why is this searching the metadata, shouldn't it just search the text?
• what is the meaning of the td..td in the search expression?

%SEARCH{ "[S]tatus.*(td..td|value\=).*[W]aiting" casesensitive="on" type="regex" 
nosearch="on" nototal="on" format="| [[$topic]]<br /> ($date - $rev - 
[[%SCRIPTURLPATH{rdiff}%/$web/$topic][Diffs]]) |"}%

Solution

%SEARCH depends on grep, and grep searches the whole file, including the meta data.

An example meta data form field is:

%META:FIELD{name="OperatingSystem" title="OperatingSystem" value="OsWin"}%

So a search for a form field could look like:

%SEARCH{ "[O]peratingSystem.*value\=.*[O]sWin" type="regex" ... }%

• Using square brackets is a trick to avoid a hit on the topic doing the search.
• The .* indicate that there can be any number of characters between OperatingSystem and value in the (whole) file

Now the original file format of the category table (the predecessor of the TWiki forms) looks like this:

<td valign="top" align="right"> OperatingSystem:  </td><td>  OsWin </td>

The following search finds topics in the old and new format:

%SEARCH{ "[O]peratingSystem.*(td..td|value\=).*[O]sWin" type="regex" ... }%

The td..td matches td<>td; a simple search on "[O]peratingSystem.*[O]sWin" could find a hit in the topic text by coincidence.

A simple %SEARCH{ "[O]peratingSystem.*value\=.*[O]sWin" ...}% search is sufficient if you do not have topics in the old format.

Pattern 10: Search all topics that have been moved

Problem

How would I go about listing all moved topics ?

Solution

Search for the META:TOPICMOVED meta data. Type this:

Moved topics: %SEARCH{ "%META\:TOPICMOVED" type="regex" format="$topic, " nosearch="on" noheader="on" nosummary="on" }%

to get this (limited to 10 results):

Moved topics: ATasteOfTWikiTemplate, DeleteOrRenameAnAttachment, MaintenanceTopics, ManagingUsers, OneYearSiteStatistics, PatternSkinPalette, ShorterUrlCookbook, SpecifyingConfigurationItemsForExtensions, SupplementalDocument, TWikiCodeOfConduct,

Related Topics: UserDocumentationCategory, SearchHelp, TWikiVariables, FormattedSearch, RegularExpression

-- Contributors: TWiki:Main.AntonAylward, TWiki:Main.ArthurClemens, TWiki:Main.JosMaccabiani, TWiki:Main.PeterThoeny, TWiki:Main.SueLocke

---

Comments & Questions about this Distribution Document Topic

Note RegisterCgiScriptRewrite deprecates the use of bullet fields for user details. User details are moved into form data in DevelopBranch so that they can be more readily manipulated.

-- MartinCleaver - 22 Jul 2005

Included info from SearchDocumentation (pattern 7) and ListAllMovedTopics (pattern 8)

-- JosMaccabiani - 03 Sep 2005

Is CategoryCookbook still in use? The DocsAdminForm doesn't have a related topics field...

-- JosMaccabiani - 30 Oct 2005

The assumption that "%SEARCH depends on grep, and grep searches the whole file, including the meta data" assumes that the storage is file-oriented. This may not hold in future releases. In addition, future releases may move metadata out of band for performance reasons.

Searching metdata should use %METASEARCH instead.

-- AntonAylward - 31 Oct 2005

Hmmm... Pattern 3a doesn't work on Dakar anymore, because of the form-based user pages. Who knows an alternative? (e.g. what does the regex for the formatted search on TWikiUsers look like?

-- JosMaccabiani - 18 May 2006

If all your users are in forms, you should be able to use $formfield to pull out this values, i think.

-- CrawfordCurrie - 18 May 2006

Thanks CC, I updated pattern 3.

-- JosMaccabiani - 19 May 2006

Thanks CC, I updated pattern 3. I knicked it from UserList

Who knows the regex to pull out the user names that belong to a certain Group? That would be great for Application-based user lists.

-- JosMaccabiani - 19 May 2006

Updated example pattern 4 (parent) and new pattern 6 (bullet list).

-- ArthurClemens - 08 Dec 2006

I've been using [f]ieldname.*?value=[f]ieldvalue to pull out metaform values. I assumed the inclusion of the ? before value would give me the value statement for the correct fieldname, while leaving it out would give me the value statement for the last fieldname on the page. If this isn't the case, could someone explain it to me? Thanks in advance.

-- MatthewCardozo - 07 Feb 2007

.*? is a "non-greedy" scan to whatever follows next; .* is a "greedy" scan. The non-greedy scan finds the first occurance, the "greedy" scan the last occurance on the same line. In your case there is little difference, except if the value itself has a value= pattern.

These types of questions are best handled in the Support web.

-- PeterThoeny - 07 Feb 2007

How used %SEARCH{"TOPICPARENT.*name\=.*%BASETOPIC%" web="Main" type="regex" scope="text" format="$topic"}% To return me the value from PARENT of files?

-- AndreMatos - 19 Apr 2007

These types of questions are best handled in the Support web.

-- PeterThoeny - 20 Apr 2007

Some advice that might help anyone struggling with searching for form data; this drove me crazy as it was not clear from any of the documentation.

If you have a field with a non-Wiki name, you need to strip out ALL the non-alphanumeric charcters when you include the name of the field in the meta search. Note that "ALL alphanumeric" means no spaces, no question marks, nothing.

For example, I had a field on a form called "What would you most like to improve in the department?". If you put this in verbatim in the query, you will get no results. You have to refer to the field as Whatwouldyoumostliketoimproveinthedepartment. Then it works fine.

%SEARCH{ "^%%META:FORM\{name=.ProfileForm" nosearch="on" type="regex" format="|[[$topic][$formfield(Name)]] | $formfield(Whatwouldyoumostliketoimproveinthedepartment) | "}%

Hope this helps!

-- JosephDaSilva - 01 Feb 2008

Pattern 3, Solution 2, specifying the word MULTIPLE may allow two or more names to be selected, but I don't think the resulting "view" can make use of this. I think the view is called with URL parameters ?topic=Fred&topic=Joe It would be nice if the example showed how you can access both Fred and Joe, as it were...

-- GrahamWatson - 2009-07-14

I am new to TWiki. Could anyone give idea on searching EDITTABLE data? Nowhere in TWiki document mentioned for searching and displaying the ediitable data on another topic

-- RathnaKalanjiam - 2010-04-28

Please ask support questions in the Support web. This section is about feedback on the documentation of this page.

-- PeterThoeny - 2010-04-28

Please use the Support forum if you have questions about TWiki features. This comment section is about the documentation of this topic.