FormQueryPlugin

The FormQueryPlugin provides a query interface to the TWiki::Contrib::DBCacheContrib module, giving a way to present query results as formatted tables.

When using the plugin each topic is treated as a container for data fields. These fields can be the fields in the form attached to the topic, or can be parsed out of the topic. The database is managed by the TWiki::Contrib::DBCacheContrib module, which must be installed for this plugin to work.

The tutorial section of this topic gives an introduction to formulating and formatting queries using this plugin, and is a good place to start.

Note: This variant of FormQueryPlugin is incompatible with earlier releases by

• removing support for features that are not part of the core functionality of this plugin, and
• striving towards compatibility with core TWiki functionality and key plugins such as SpreadSheetPlugin.

If the requirements management features of earlier versions of FormQueryPlugin are required, do not use this version.

Features

• Perform complex queries on the database
• Extensive conditional formatting support
• Automatic derivation of new topic names
• Recognition and special formatting of key data strings

How the database gets built

The basic database is built using the DBCacheContrib. Refer to the documentation of that module for details of how it works.

As well as the form data loaded by the DBCacheContrib, the FormQueryPlugin adds the ability to extract embedded tables from topics.

Embedded tables that follow a specific format (defined using the EditTablePlugin) are added to the topic as an array of rows, each of which contains a map keyed on the fields and mapping to the values in the table. This array is added to the topic as a field named the same as the table type. Where there are multiple tables of the same type in a topic, they are concatenated together into a single array. See Settings for more information about how to set up tables.

Topics can also be automatically related to each other by name. The plugin uses something called Topic Relations that allow the creation of two-way relationships between topics. Topic relations from parent topics to child topics are stored in the map of the parent topic using the name of the relation as the field name and an array of references to the maps of child topics. Reverse relations from child to parent topics are also stored. See Settings for more information about how to set up topic relations.

Most of the rest of this documentation describes how to search this database and generate reports.

Syntax Rules

A query has first to be formed using FORMQUERY, and then shown using SHOWQUERY. You can combine these operations using DOQUERY. Queries are output according to a format specification in the parameters to SHOWQUERY. You can cache these specifications for re-use by defining them using TABLEFORMAT.

Queries

%FORMQUERY

%FORMQUERY is the basic query mechanism. A %FORMQUERY works either on the whole database or, if the query parameter is given, the results of another query.

Parameter	Description
name	Required, gives the name of the query result for use in %SHOWQUERY or another %FORMQUERY
search	Required, the search to perform (see Search operators). Default parameter; the keyword search may be omitted.
casesensitive	Optional, if missing or off search is not casesensitive.
query	Optional, the name of the query to refine. If missing, defaults to the whole database
extract	Optional, the name of a field in each of the matched maps to flatten out into the returned list.
moan	Optional, "on" or "off", if set to "off" will disable match failed message

For example,

%FORMQUERY{name=AQuery search="Owner='Main\.Fred'"}%
%FORMQUERY{name=BQuery query=AQuery search="Product='Boiled Egg'"}%
%FORMQUERY{name=CQuery query=BQuery search="" extract=CookingTimes}%
%FORMQUERY{name=DQuery query=CQuery search="Time < '4'"}%

will search for all topics with a form field Owner set to Main.Fred, then filter that down to those topics that have Product set to Boiled Egg. Then it will extract and flatten out the embedded table CookingTimes in each matched topic. By "flatten out" we mean that future queries on CQuery must refer to the fields of the CookingTimes table, not the fields of the topic, i.e. CQuery will be an array of all the rows in the embedded table. Finally it will filter down to those rows that have the column Time < 4 minutes.

Of course there is more than one way to skin a cat. A faster way to formulate the same query would be to say:

%FORMQUERY{name=AQuery search="Owner='Main\.Fred' AND Product='Boiled Egg'" extract=CookingTimes}%
%FORMQUERY{name=DQuery query=AQuery search="Time < '4'"}%

FORMQUERY is normally silent i.e. it generates no output unless there is an error, in which case an error description is inserted. This error message can be disabled using the moan parameter which makes FORMQUERY totally silent.

Search operators

Fields are given by name, and values by strings or numbers. Strings should always be surrounded by 'single-quotes'. Strings which are regular expressions (RHS of , ~ operators) use 'perl' regular expression syntax (google for =perlre for help). Numbers can be signed integers or decimals. Single quotes in values may be escaped using backslash (\).

The following operators are available:

Operator	Result	Meaning
=	Boolean	LHS exactly matches the regular expression on the RHS. The expression must match the whole string.
!=	Boolean	Inverse of =
=~	Boolean	LHS contains RHS i.e. the RHS is found somewhere in the field value.
<	Boolean	Numeric <
>	Boolean	Numeric >
>=	Boolean	Numeric >=
<=	Boolean	Numeric <=
lc	String	Unary lower case
uc	String	Unary UPPER CASE
EARLIER_THAN	BOOLEAN	Date is earlier than the given date
LATER_THAN	Boolean	LHS is later than the given date (string containing a date e.g. '1 Apr 2003')
WITHIN_DAYS	Boolean	Date (which must be in the future) is within n working days of todays date
!	Boolean	Unary NOT
AND	Boolean	AND
OR	Boolean	OR
()	any	Bracketed subexpression

Dates for EARLIER_THAN, LATER_THAN and WITHIN_DAYS must be dates in the format expected by Time::ParseDate (like the ActionTrackerPlugin). WITHIN_DAYS works out the number of working days (i.e. excluding Saturday and Sunday). Apologies in advance if your weekend is offset ± a day! Integers will automatically be converted to dates, by assuming they represent a number of seconds since midnight GMT on 1st January 1970.

Search operators work on the fields of each map, be it a topic or an embedded table row.

Fields can be simple field names or can be more complex, and may even contain embedded searches. See Fields below for more information.

A search defined as some text string without any search operators is interpreted as text=~'string'.

At present there is no way to constrain a search on the contents of a contained table, such as an embedded table or forward relation. However there are usually ways around this; for example, a %FORMQUERY that uses extract to flatten all the sub-tables, and then use of the parent relation to complete the search. For example:

%FORMQUERY{name=AQuery search="Product='Boiled Egg'" extract=CookingTimes}%
%FORMQUERY{name=DQuery query=AQuery search="Time < '4' AND Plugins.CookingTimes_of.Owner='Main\.Fred'"}%

Fields

Field references can be as simple as the name of a field, or may be more complex expressions that can even include embedded searches. The precise interpretation of the syntax depends on the type (Map or Array) of the object being referenced.

Arrays

• N where N is a number will get the Nth entry in the array e.g. attachments.9
• X will return the sum of the subfield X of each entry e.g. TaskTable.Effort will sum the Effort column in a table called TaskTable.
• [?search] will perform the given search over the entries in the array. Always returns an array result, even when there is only one result. For example: attachments[?name='pitcha.gif'] will return an array of all the entries that have their subfield name set to pitcha.gif.
• [*X] will get a new array made from field X of each entry in this array. For example attachments[*size] will get an array of the sizes of each attachment.

Maps

• X will get the subfield named X. For example, Product will get the formfield named Product.
• X.Y will get the subfield Y of the subfield named X.

In all cases

• # means "reset to root". So #.Y will return the subfield Y of the Map at the root of the query.

Reports

%TABLEFORMAT

%TABLEFORMAT provides a template for the display of values extracted from a query (like the old Fortran 'format' statement).

Parameter	Description
name	Required, gives the name of the format for use in %SHOWQUERY or another %FORMQUERY
format	Required, gives the format of the body rows (or the format to subclass)
header	Optional, gives the format of the header row
separator	Optional, separate rows with this (default newline)
footer	Optional, gives the string to use to terminate the table
sort	Optional, defines the sort order
help	Optional, if present then a helpful message will be output for undefined fields. So if you see a table entry that is blank and you don;t think it should be, add the help parameter and it should give some guidance.
moan	Optional, "on" or "off", if set to "off" disables error output

The header, footer and format parameters are very similar in application to the parameters used in the standard %SEARCH TWiki command. For the uninitiated, the header is used to define the first row of the results table:

header="| *Topic* | *Summary* |"
header="<table><tr><th>Topic</th><th>Summary</th></tr> |"
footer="</table>"

The format parameter is overloaded so it can also be used to refer to a table format previously defined using %TABLEFORMAT. This allows you to derive new formats (e.g. with different sort orders) For example,

%TABLEFORMAT{ name=TBLFMT format="|$topic|" sort="Owner,Priority"}%
%TABLEFORMAT{ name=SUBFMT format=TBLFMT sort="Priority,Owner" }%

Any of the header, footer or sort parameters in the referenced table format can be overridden by giving new values in the %SHOWQUERY.

The footer is usually only used for tables generated directly in HTML, in which case it is usually set to footer="</table>"

footer="</table>"

The format is used to define the rows in the body of the table:

format="| $topic | $Summary |"

Any fields in the form in the topic can be output by putting a $ sign before the name of the field. The two special fields $topic and $form are available for outputting the name of the topic and the type of the form. You can also refer to fields in related topics using the syntax $field.field where $field is the name of the relation and field is a field in the related topic; for example, format="|$Child_of.topic|".

The following special variables can be used in the format string:

Name:	Expands To:
$web	Name of the web
$topic	Topic name
$topic(20)	Topic name, "- " hyphenated each 20 characters
$topic(30, -<br />)	Topic name, hyphenated each 30 characters with separator "-<br />"
$topic(40, ...)	Topic name, shortended to 40 characters with "..." indication
$parent	Name of parent topic; empty if not set
$parent(20)	Name of parent topic, same hyphenation/shortening like $topic()
$text	Formatted topic text.
$locked	LOCKED flag (if any)
$date	Time stamp of last topic update, e.g. 06 Feb 2026 - 12:39
$isodate	Time stamp of last topic update, e.g. 2026-02-06T12:39Z
$rev	Number of last topic revision, e.g. 1.4
$username	Login name of last topic update, e.g. jsmith
$wikiname	Wiki user name of last topic update, e.g. JohnSmith
$wikiusername	Wiki user name of last topic update, like Main.JohnSmith
$createdate	Time stamp of topic revision 1.1
$createusername	Login name of topic revision 1.1, e.g. jsmith
$createwikiname	Wiki user name of topic revision 1.1, e.g. JohnSmith
$createwikiusername	Wiki user name of topic revision 1.1, e.g. Main.JohnSmith
$summary	Topic summary, with 162 characters
$summary(50)	Topic summary, with 50 characters
$summary(showvarnames)	Topic summary, with %ALLTWIKI{...}% variables shown as ALLTWIKI{...}
$summary(noheader)	Topic summary, with leading ---+ headers removed Note: The tokens can be combined into a comma separated list like $summary(100, showvarnames, noheader)
$formname	The name of the form attached to the topic; empty if none
$formfield(name)	The field value of a form field; for example, $formfield(TopicClassification) would get expanded to PublicFAQ. This applies only to topics that have a TWikiForm. An alternative to using the name of the field directly.
$formfield(name, 10)	Form field value, "- " hyphenated each 10 characters
$formfield(name, 20, -<br />)	Form field value, hyphenated each 20 characters with separator "-<br />"
$formfield(name, 30, ...)	Form field value, shortended to 30 characters with "..." indication
$pattern(reg-exp)	A regular expression pattern to extract some text from the topic text (does not search meta data; use fields instead). • Specify a RegularExpression that covers the whole text (topic or line), which typically starts with .*, and must end in .* • Put text you want to keep in parenthesis, like $pattern(.*?(from here.*?to here).*) • Example: $pattern(.*?\*.*?Email\:\s*([^\n\r]+).*) extracts the email address from a bullet of format * Email: ... • This example has non-greedy .*? patterns to scan for the first occurance of the Email bullet; use greedy .* patterns to scan for the last occurance • Limitation: Do not use .*) inside the pattern, e.g. $pattern(.*foo(.*)bar.*) does not work, but $pattern(.*foo(.*?)bar.*) does • Note: Make sure that the integrity of a web page is not compromised; for example, if you include an HTML table make sure to include everything including the table end tag
$n or $n()	New line
$nop or $nop()	Is a "no operation". This variable gets removed; useful for nested search
$quot	Double quote ("). Alternatively, write \" to escape it.
$percnt	Percent sign (%)
$dollar	Dollar sign ($)

It is also possible to expand the contents of tables embedded in the topic, in two ways:

• using the syntax $field[format=FORMAT] where $field is the table type, which is named the same as the table topic, and FORMAT is the name of a format previously defined using %TABLEFORMAT and suitable for expanding the sub-table.
• using the syntax $field.subfield where subfield is a numeric field in the contents of the embedded table (i.e. it contains a number). In this case the result will be the sum of all subfield values over all the rows in the table.

sort is used to define the sort order, and is a list of field names (without $'s) separated by commas. You can reverse the sort order for a column by prepending '-' to the field name. For example,

sort="Owner,-Priority"

The default sort method is string comparison. If the field data is numeric, then you can put a '#' in front of the field name to force the sort to be numeric. For example,

sort="Owner,#Priority"

or

sort="Owner,-#Priority"

%TABLEFORMAT is silent i.e. it produces no output on the page unless there is an error, in which case a descriptive error message is output. This message can be disabled using the moan parameter.

%SHOWQUERY

Used to show the results of a query, %SHOWQUERY accepts all the same parameters as %TABLEFORMAT (except name), and also the following parameters:

Parameter	Description
query	Required, gives the name of the query to show
start	Optional, first row to display
limit	Optional, number of rows to display, or all

For example,

%SHOWQUERY{ query=AFORMQUERY format=ATBLFMT }%

Note: If you render your results in TWiki table format and limit the rows displayed using limit (e.g. first 10 from 99 rows), and also use TablePlugin to allow sorting on the headers, then only the 10 displayed rows will be sorted by TablePlugin (and not the full 99 rows in the table).

%DOQUERY

Combines a query and showing the results of that query, %DOQUERY accepts all the same parameters as %FORMQUERY and %SHOWQUERY (except name and query).

Interface to SpreadSheetPlugin

%QUERYTOCALC

Used to transform the results of a query into a format which SpreadSheetPlugin can operate on. %QUERYTOCALC accepts all the same parameters as %TABLEFORMAT (except name), and also the following parameters:

Parameter	Description
query	Required, gives the name of the query to show
start	Optional, first row to display
limit	Optional, number of rows to display, or all

For example,

%QUERYTOCALC{ query=AFORMQUERY format=ATBLFMT }%

The query must be formatted as a table. This variable produces no output, but must precede any calculation on the resultant query.

%SHOWCALC

Operates on the result of a query represented as a table, using a previous %QUERYTOCALC. Accepts a single, unnamed parameter:

Parameter	Description
	Required; the calculation to perform

The parameter is text as required by the $CALC statement (see SpreadSheetPlugins); all operations supported by %CALC are supported by %SHOWCALC. This command behaves identical to %CALC when used following a table; it operates on the implicit table constructed by a preceding %QUERYTOCALC statement. Arbitrary many %SHOWCALC can be performed; all apply to the preceeding implicit table.

Example:

%SHOWCALC{"$ROW()"}%

returns the number of matches of a query that had been formatted as an implicit table.

Querying results of a search

%SUMFIELD

Used to calculate the sum of a numeric field across all the matches in a query result.

Parameter	Description
query	Required, name of the query to sum over
field	Required, name of the field to sum

For example, let's say we have a form type called "VitalStatistics" that has a field "Age" in it. We can sum all the ages in all occurences of the form using:

%FORMQUERY{name=vs search="form='VitalStatistics'"}%
Sum of ages = %SUMFIELD{query=vs field="Age"}%

The result is a single number representing the sum of all the fields.

Note that values embedded in tables can be summed using the shorthand described above.

%MATCHCOUNT

Used to determine the total number of results a query returned.

Parameter	Description
query	Required, name of the query to sum over

The result is a single number representing the number of results the query returned.

Tutorial

See TWiki::Plugins/FormQueryPluginTutorial for a tutorial describing how to use the plugin in real-world applications.

Formatted query examples

%TABLEFORMAT% is able to generate (roughly) the same output as FormattedSearch, based on an earlier query. Multiple different formatted results can be produced without having to rerun the query.

Bullet list showing topic name and summary

Write this:

%FORMQUERY{name="q1" search="topic=~'FAQ'" web="TWiki"}%
%SHOWQUERY{query="q1" header="   * *Topic: Summary:*" format="   * [[$web.$topic]]: $summary(25)"}%

To get this:

%FORMQUERY{name="q1" search="topic=~'FAQ'" web="TWiki"}% %SHOWQUERY{query="q1" header=" * *Topic: Summary:*" format=" * $web.$topic: $summary(25)"}%

Extract some text from a topic using regular expression

Write this:

%FORMQUERY{name="q1" search="text=~'__Back to:__ TWikiFAQ'" web="TWiki"}%
%SHOWQUERY{query="q1" header="TWiki FAQs:" format="   * $pattern(.*?FAQ\:[\n\r]*([^\n\r]+).*) [[$web.$topic][Answer...]]"}%

To get this:

%FORMQUERY{name="q1" search="text=~'__Back to:__ TWikiFAQ'" web="TWiki"}% %SHOWQUERY{query="q1" header="TWiki FAQs:" format=" * $pattern(.*?FAQ\:[\n\r]*([^\n\r]+).*) Answer..."}%

Nested Search

Search can be nested. For example, search for some topics, then form a new search for each topic found in the first search. The idea is to build the nested search string using a formatted search in the first search.

For example, let's search for all topics that contain the word "culture" (first search), and let's find out where each topic found is linked from (second search).

• First search:
  • %FORMQUERY{name="q1" search="text=~'culture'" web="TWiki"}% %SHOWQUERY{query="q1" format="   * $web.$topic is referenced by:$n(list all references)"}%
• Second search. For each hit we want this search:
  • %DOQUERY{"(topic found in first search)" separator=", " web="TWiki" format="$web.$topic"}%
• Now let's nest the two. We need to escape the second search, e.g. the first search will build a valid second search string. Note that we escape the second search so that it does not get evaluated prematurely by the first search:
  • Use $percnt to escape the leading percent of the second search
  • Use \" to escape the double quotes
  • Use $dollar to escape the $ of $topic

Write this:

%FORMQUERY{name="q1" search="text=~'culture'" web="TWiki"}%
%SHOWQUERY{query="q1" format="   * $web.$topic is referenced by:$n      * $percntDOQUERY{\"$topic\" separator=\", \" web=\"TWiki\" format=\"$dollarweb.$dollartopic\"}$percnt "}%

To get this:

%FORMQUERY{name="q1" search="text=~'culture'" web="TWiki"}% %SHOWQUERY{query="q1" format=" * $web.$topic is referenced by:$n * $percntDOQUERY{\"$topic\" separator=\", \" web=\"TWiki\" format=\"$dollarweb.$dollartopic\"}$percnt "}%

Note: Nested search can be slow, especially if you nest more then 3 times. Nesting is limited to 16 levels. For each new nesting level you need to "escape the escapes", e.g. write $dollarpercntDOQUERY for level three, $dollardollarpercntDOQUERY for level four, etc.

Most recently changed pages

Write this:

%FORMQUERY{name="q1" search="" web="TWiki"}%
%SHOWQUERY{query="q1" limit="7" sort="info.date" format="| $web.$topic | $wikiusername | $date |"}%

To get this:

%FORMQUERY{name="q1" search="" web="TWiki"}% %SHOWQUERY{query="q1" limit="7" sort="info.date" format="| $web.$topic | $wikiusername | $date |"}%

Search with conditional output

A regular expression search is flexible, but there are limitations. For example, you cannot show all topics that are up to exactly one week old, or create a report that shows all records with invalid form fields or fields within a certain range, etc. You need some additional logic to format output based on a condition:

1. Specify a search which returns more hits then you need
2. For each search hit apply a spreadsheet formula to determine if the hit is needed
3. If needed, format and output the result
4. Else supress the search hit

This requires the TWiki:Plugins.SpreadSheetPlugin. The following example shows all topics that are up to exactly one week old.

Write this:

%CALC{$SET(weekold, $TIMEADD($TIME(), -7, day))}%
%FORMQUERY{name="q1" search="name='.*'" web="TWiki"}%
%SHOWQUERY{query="q1" sort="info.date" format="$percntCALC{$dollarIF($dollarTIME($date) < $dollarGET(weekold), , | $web.$topic | $wikiusername | $date | $rev |)}$percnt" }%

• The first line sets the weekold variable to the serialized date of exactly one week ago
• The SHOWQUERY has a deferred CALC. The $percnt makes sure that the CALC gets executed once for each search hit
• The CALC compares the date of the topic with the weekold date
• If topic is older, a <nop> is returned, which gets removed at the end of the TWiki rendering process
• Otherwise, the search hit is formatted and returned

To get this:

%FORMQUERY{name="q1" search="name='.*'" web="TWiki"}% %SHOWQUERY{query="q1" sort="info.date" format="$percntCALC{$dollarIF($dollarTIME($date) < $dollarGET(weekold), , | $web.$topic | $wikiusername | $date | $rev |)}$percnt" }%

Note: If you don't see any output, then there are no files newer than 7 days. Rows without entries in all but the first field are due to incorrect topics present in the TWiki web (no topic info line, thus we cannot find date, author, rev).

Embedding search forms to return a formatted result

Use an HTML form and an embedded formatted search on the same topic. You can link them together with an %URLPARAM{"..."}% variable. Example:

Write this:

<form action="%SCRIPTURLPATH%/view%SCRIPTSUFFIX%/%WEB%/%TOPIC%">
Find Topics: 
<input type="text" name="q" size="32" value="%URLPARAM{"q"}%" />
<input type="submit" value="Search" />
</form>
Result:

%FORMQUERY{name="q1" search="text=~''" web="TWiki"}%
%SHOWQUERY{query="q1" limit="7" format="   * $web.$topic: %BR% $summary" }%

To get this:

Find Topics:

Result (first 7): %FORMQUERY{name="q1" search="text=~''" web="TWiki"}% %SHOWQUERY{query="q1" limit="7" format=" * $web.$topic:
$summary" }%

Generate a table with embedded calculations

Assume you want to produce a summary of all the large attachments in a topic (where by large we mean any attachment greater than 10k). The table should show topic, the name of the attachment, the file size in k, and the date when the attachment was created.

Write this:

%FORMQUERY{name="all" search="" web="TWiki" extract="attachments"}%
%FORMQUERY{name="big" query="all" search="size>'10240'"}%
%SHOWQUERY{query="big" format="|$_up.web.$_up.name|$name|  $percntCALC{$dollarROUND($dollarEVAL($size / 1024), 1)k}$percnt|$percntCALC{$dollarFORMATTIME($date, $dollarday $dollarmon $dollaryear)}$percnt|"}%

To get this:

%FORMQUERY{name="all" search="" web="TWiki" extract="attachments"}% %FORMQUERY{name="big" query="all" search="size>'10240'"}% %SHOWQUERY{query="big" format="|$_up.web.$_up.name|$name| $percntCALC{$dollarROUND($dollarEVAL($size / 1024), 1)k}$percnt|$percntCALC{$dollarFORMATTIME($date, $dollarday $dollarmon $dollaryear)}$percnt|"}%

Topic Relations

Topic relations are defined by setting the FORMQUERYPLUGIN_RELATIONS variable in the WebPreferences topic of the web to be processed. This defines a semicolon-separated list of rules for deriving relationships between topics using common portions of names. This is best illustrated by an example. Let's say we set FORMQUERYPLUGIN_RELATIONS to be:

Plugins.ReQ%Ax%B Plugins.SubReq Plugins.ReQ%A; Plugins.TiT%An%B Plugins.TestItem Plugins.ReQ%A

This describes two rules, the first of which says "Any topic named "ReQ<something>x<something>" is a SubReq of another topic called "ReQ<something>". Single upper case characters with a preceding percent sign are used to define the <something>s. So, in this example, ReQ1456 has the SubReq relation to ReQ1456x7. The reverse relation is also inserted, but appending _of to the relation name, so ReQ1456x7 has the SubReq_of relation to ReQ1456.

Obviously a single topic may have many topics that have the same relation to it (i.e. they are one-to-many), so forward relations are stored as arrays in the maps that represent topics. Reverse relations are many_to_one so they are stored as a simple field in the other topic. Relations are stored as pointers to the maps of the other topic so they can't be printed as if they were simple names. Instead you have to use them to refer to fields within the other topic. For example, in a %SHOWQUERY showing ReQReQ1456x7 you can't refer simply to $SubReq_of to get ReQ1456 (the name of the related topic); you have to use $SubReq_of.topic instead.

The default value of FORMQUERYPLUGIN_RELATIONS is none.

Tables

Tables to be read into the database must use the EditTablePlugin 'include= topic ' feature (or the EditTablerow 'template= topic ' feature), which define the format of the tables in another topic. Tables to be read are defined by setting the FORMQUERYPLUGIN_TABLES variable in the WebPreferences topic of the web. This is a list of the names of the topics that EditTablePlugin or EditTablerowPlugin use to find the table formats. Tables are read in as arrays of maps, with each map keyed on a field name generated by stripping all non-alphabetic characters from the column header. Note that the column header where the table is defined is used, not the column header in the EditTable include= topic. If the FORMQUERYPLUGIN_TABLES variable is set to All (case-insensitive), then any valid table is added to the database.

The defaut is to load all tables:

• Set TABLES = all

Two special map entries are added to each table row, topic and the equivalent of the reverse relation described above, represented by _up. So if you load a table type "CookingTimes", each row will have topic set to the name of the topic it was loaded from and _up set to the map of that topic. This allows you to refer to the fields of the topic that contains a table even after the table has been extracted.

When a table contains calculations using TWiki:Plugins.SpreadSheetPlugin, these are applied before the map entries are created for this table. These calculations must not refer to data outside of the table.

If there are multiple EDITTABLE calls with the same table type, all the table data is concatenated into a single table.

The current value of FORMQUERYPLUGIN_TABLES is %FORMQUERYPLUGIN_TABLES%.

For example, let's say we have a table in a topic called TableDefinition, like this:

Name	Description

Now, in another topic, TableUse, we have

%EDITTABLE{include="TableDefinition"}%
| Cat | Small furry animal |
...
%EDITTABLE{include="TableDefinition"}%
| Dog | Latch used to seal a hatch on a ship |

In this case, $TableDefinition.0.Name is Cat, $TableDefinition.1.Name is Dog.

Plugin Installation Instructions

You do not need to install anything in the browser to use this extension. The following instructions are for the administrator who installs the extension on the server where TWiki is running.

Like many other TWiki extensions, this module is shipped with a fully automatic installer script written using the BuildContrib.

• If you have TWiki 4.1 or later, you can install from the configure interface (Go to Plugins->Find More Extensions)
  • The webserver user has to have permission to write to all areas of your installation for this to work.
• If you have a permanent connection to the internet, you are recommended to use the automatic installer script
  • Just download the FormQueryPlugin_installer perl script and run it.
• Notes:
  • The installer script will:
    • Automatically resolve dependencies,
    • Copy files into the right places in your local install (even if you have renamed data directories),
    • check in new versions of any installed files that have existing RCS histories files in your existing install (such as topics).
    • If the $TWIKI_PACKAGES environment variable is set to point to a directory, the installer will try to get archives from there. Otherwise it will try to download from twiki.org or cpan.org, as appropriate.
    • (Developers only: the script will look for twikiplugins/FormQueryPlugin/FormQueryPlugin.tgz before downloading from TWiki.org)
  • If you don't have a permanent connection, you can still use the automatic installer, by downloading all required TWiki archives to a local directory.
    • Point the environment variable $TWIKI_PACKAGES to this directory, and the installer script will look there first for required TWiki packages.
      • $TWIKI_PACKAGES is actually a path; you can list several directories separated by :
    • If you are behind a firewall that blocks access to CPAN, you can pre-install the required CPAN libraries, as described at http://twiki.org/cgi-bin/view/TWiki/HowToInstallCpanModules
• If you don't want to use the installer script, or have problems on your platform (e.g. you don't have Perl 5.8), then you can still install manually:
  1. Download and unpack one of the .zip or .tgz archives to a temporary directory.
  2. Manually copy the contents across to the relevant places in your TWiki installation.
  3. Check in any installed files that have existing ,v files in your existing install (take care not to lock the files when you check in)
  4. Manually edit LocalSite.cfg to set any configuration variables.
  5. Run configure and enable the module, if it is a plugin.
  6. Repeat from step 1 for any missing dependencies.

• The variable %FORMQUERYPLUGIN_ENABLE% must be set either globally or in the web where you want to use it. If the plugin is enabled globally (on all webs), this may have an unwelcome performance impact, as the plugin may create a cache of topics in a web where it is not intended to be used. To enable the plugin for selected webs only, put the following line into the WebPreferences topic of the web you want to enable it on:

      * Set FORMQUERYPLUGIN_ENABLE = 1

• You can set a global default for the moan parameter by setting the =FORMQUERYPLUGIN_MOAN variable in TWikiPreferences, WebPreferences in individual webs, or in individual topics.
• FormQueryPlugin must appear in the Plugins evaluation order before TWiki:Plugins/SpreadSheetPlugin. You can ensure this by putting FormQueryPlugin immediately after DefaultPlugin in the {PluginsOrder} list in configure.
• Note: If you are using it, TWiki:Plugins/MacrosPlugin must appear in the Plugins evaluation order before FormQueryPlugin.

---

Plugin Info

Plugin Author:	TWiki:Main/CrawfordCurrie, TWiki:Main/ThomasWeigert
Plugin Version:	0 (21 May 2007)
Change History:
21 May 2007	Bugs:Item3969 - 8bit email fix (TWiki:Main.WillNorris)
24 Apr 2007 (13527)	Merge YetAnotherFormQueryPlugin back in. Fixed a lot. Prepared for use with TWiki 4.2.
04 Sep 2006	Expand calculations using TWiki:Plugins.FormQueryPlugin.
23 Jun 2006	Ported to Dakar.
9 Jul 2005	Added support for tables defined using TWiki:Plugins/EditTablerowPlugin
	Added support for searches over multiple webs
	Support %MATCHCOUNT%
	Allow TWiki:Plugins/SpreadSheetPlugin computation to be applied to the result of a query
	Made "moan" a preference rather than a per call option
	Removed %TOPICCREATOR%
	Removed %ARITHMETIC% (as one can use the TWiki:Plugins/SpreadSheetPlugin)
	Removed the color map feature
	Extended %FQPINFO% to support showing results of queries
	Integrate the Attribute parser from TWiki:Codev/DakarRelease
	Add caseinsensitivity as option
	Handle one line at a time so that query memory can be reused in another query
	Added the special variables supported by FormattedSearch
	Changed some of the option names to be more consistent with TWikiSearch (e.g., row_count)
	More flexibility for using tables
	Add a %DOANDSHOWQUERY% tag
	Support embedding of %SEARCH% and %FORMQUERY% in the format option to allow searches and queries to be applied to the result of a query (a poor man's intersection)
	Added option to switch on/off "Undefined" messages for undefined field values
	Expanded fields no longer have spaces around them. This is more correct, but requires care in format definitions.
16 Jul 2004	Split into sub-components
02 Apr 2004	TWiki:Main/SimonHardyFrancis bugfixes incorporated. TWiki:Main/TimSlidel patches incoporated for mod_perl, autocreate, and collapsing same-type tables in a single topic
05 Feb 2004	Bugfix version
11 Jul 2003	Initial version
TWiki Dependencies:	TWiki:Plugins.SpreadSheetPlugin
Dependencies:	Name Version Description TWiki::Contrib::DBCacheContrib >=1.001 Required. Download and install from the TWiki:Plugins/DBCacheContrib code library. Time::ParseDate Required. Available from CPAN. Storable >=2.07 2.13 recommended; accelerates cache handling. Available from CPAN TWiki::Plugins::MacrosPlugin >=1.000 Recommended; used for parameterised includes. Download from TWiki:Plugins/MacrosPlugin repository.
Perl Version:	5.0
Plugin Home:	http://TWiki.org/cgi-bin/view/Plugins/FormQueryPlugin
Feedback:	http://TWiki.org/cgi-bin/view/Plugins/FormQueryPluginDev
Demo URL:	http://www.owiki.org/FQP/

Copyright

This code is based on an original development of Motorola Inc. and is protected by the following copyrights:

• Copyright © 2002-2003, 2005 Motorola. All Rights Reserved.
• Copyright © 2004. Crawford Currie http://www.c-dot.co.uk

Extending or customising

Extension or customisation is welcome, as long as all extensions are described and code provided back to the author.

The module is shipped with a perl build file, which should be used for installation and testing. Testing is done using Test::Unit, and is invoked using the 'test' build target. Writing tests is a useful way of feeding back bugs as well. I can't encourage you enough to maintain and extend the tests!

License

As required for the publication of all extensions to TWiki, this software is published under the terms of the GNU General Public License.

This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details, published at http://www.gnu.org/copyleft/gpl.html

-- TWiki:Main/ThomasWeigert - 04 Sep 2006

---

I recently implemented a layer that supports using TWiki webs as a generic-ish database. I include here the full text of the documentation for your amusement. Please feel free to add remarks after the final <hr>

-- CrawfordCurrie - 25 Jun 2003

Comments, Questions, Howls of laughter:

Wow! This looks really, really cool. If you haven't seen my comments in logically nested webs then I think we're thinking along the same lines. When I was doing the big refactor of that topic at the weekend I factored out large swathes of my comments (only felt comfortable savaging my own comments), but essentially I laid out the same idea:

• Webs as relations, topics as tuples, and any "structure" (named section, metadata) of any kind in it being an attribute.

The ideas I put forward are complimentary to a number of your features I think. References are:

• Slightly better
• Meandering Description
• "TWiki as a Non-Normalised Relational Store" - Section title might make you laugh if you haven't seen it:

I'd really like to see what you've done in practice.

-- MichaelSparks - 25 Jun 2003

Like you say, Michael, we're thinking along the same lines. I started out with a specialised "Meta" object but quickly refactored down to two base objects, "Map" and "Array". I implemented the loaders on them (to load forms and tables, and even - experimentally - actions) as decorators. The final key object for this application is a table formatter (re-used from the action tracker) that generalises header= and format= style attributes. Both Map and Array implement the same interface - principally a "get" method - which works hierarchically on the objects e.g. ->get("Topic.Field.Subfield"). The worst problem was working out all the semantics of fields in Arrays and finding clean ways to express it all in TWiki tags. The cacheing is done using the Storable module off CPAN.

<soapbox>

I implemented all this stuff under a kind of duress, because an existing plugin that was focused more on the problem had been done (we need something like this desperately and don't want to invest in DOORS). The existing plugin was a hackjob, highly specific to the problem domain, and effectivelky turned a twiki web into a ghetto. I wanted to open it up again by introducing something to do the same thing in a more generically applicable way.

Many people on Codev talk about introducing more structure or control into their webs. All the time this is going on, I'm fighting exactly that on our intranet. Each time a project team builds itself a silo, an opportunity for collaboration is missed. I've even found the same topic in two project webs, discussing eseentially the same problem, but with divergent solutions. I cross-linked them, of course, but by then it was too late to stop the religious war.

Rather than finding new ways to structure information I wish we could spend a few more watts on finding ways to link information. I'd have loved it if, when I created this topic, some magic AI in the background suddenly shouted "you didn't want to do that!" and pointed out all the existing topics on this subject.

</soapbox>

-- CrawfordCurrie - 26 Jun 2003

bloody excellent!!

can you publish the source? this is just the sort of thing i have been way too lazy to implement myself, but need for the TWikiBugSystem that we use at work. if there is any way that we can get this into the base TWiki release I would love that! to me its more important than it being an optional plugin

well done indeed!

-- SvenDowideit - 27 Jun 2003

Like I said, Sven, "I have no plans to publish this plugin unless there's strong support from people willing to help support it. "

-- CrawfordCurrie - 27 Jun 2003

ok, if you look around at what i have done (and not done) here in the twiki you will see that I don't spend regular amounts of time on anything. It always comes in spurts, with large pauses when i am busy working. I am willing to help support this code, even more so - eager to use and integrate it at my work twiki - and as such will definiatly role back any changes that i would need for that.

I would also strongly push to have it turned on here, as it gives us many of the tools needed for a task traking system (i htink we need to improve the one used here for twiki releases)

-- SvenDowideit - 28 Jun 2003

Very nice! Even though I'll have to re-read it a few times to get a fuller understanding.

I did notice that I think you switched from "BoiledEggs" to "EggTimer" when you didn't mean to in one of the examples above. (I would have fixed it, but I wasn't quite 100% sure it was a mistake.)

From what I've understood so far, you haven't necessarily included all metadata in the database, but, there is no reason why you could not?

Wish I could promise to help maintain it, but I'm not quite at that stage yet.

-- RandyKramer - 29 Jun 2003

The documentation pasted above was just a snapshot of work in progress; I've corrected the egg problem. The documentation with the plugin is a lot richer now.

All metadata in the database? No. I have not included parents, because I have found them to be very random and unreliable; relations are much more flexible and reliable. I have not included the most important meta-data, revision histories, simply because of performance. I have not included links, or more importantly categories, either. Just %META:F*, and %EDITTABLES.

I started off implementing loaders as decorators, but moved away from that because of performance. However it wouldn't be that hard to move back again and add any kind of recognisable thingies from the topics you wanted; sections, links, actions have all occurred to me. I'm sure interaction with other plugins would throw up other things as well.

Now there's an idea. Implement plugin hooks that implement the loaders. A plugin to a plugin? Perhaps better to rearchitect this support into the core. And make extracting meta-data a heck of a lot more efficient, which was the initial goal of this topic.

BTW I did some benchmarks on a 2GHz lightly loaded linux/apache server with mod_perl and Storable. Loading a 2000 topic web takes ~0.6s reading topics and ~0.3s from the cache, and 1.2s to perform a complex query that visits every topic and every embedded table. Call this 1.8/1.5. Lose Storable, I get 2.8/2.2. Lose mod_perl, 5.0/4.8.

I'm rerunning the benchmarks on a heavily loaded server; I'll report those later.

-- CrawfordCurrie - 30 Jun 2003

Very very nice! I'd certainly use it. I'd certainly try to help support it - but whether you would class that as a help or a hinderance is another matter.

-- DarrylGreen - 09 Jul 2003

I'm with Daryl - I really think this is excellent, Crawford - I would certainly use it (mainly for task tracking, prioritisation, etc). I would also be in the category of "trying to help support it", although my coding skills are not brilliant in this area.

-- RossC - 10 Jul 2003

CrawfordCurrie I add my voice (mee too ) - this is excellent idea. It is a real application build over Twiki platform. I will try to install it when available.

-- PeterMasiar - 11 Jul 2003

OK... please forgive me if I'm a little wary but I have very little time to provide support for rather too many plugins. FormQueryPlugin been running here for the last couple of weeks, serving a busy requirements DB, so it seems pretty robust. It's also spawning several derivative applications that I didn't expect....

I will check into CVS, hopefully tonight; those of you serious about helping to maintain, please get a copy from there. I'd appreciate it if people also maintained the tests, which are written in Test::Unit (CPAN). Once you confirm you are set up I'll upload the front page and zip to the plugins web (i.e. release it).

Thanks!

-- CrawfordCurrie - 11 Jul 2003

OK, it's all in CVS. -- CrawfordCurrie - 13 Jul 2003

Crawford, as you know I've been taking a look at this. Before I make any support commitment I need to build a sample application with it however. (Which I've been in the process of for a few days, but is unfortunately lower down my priority queue than I'd like - nasty bug going round isn't fun)

-- MichaelSparks - 13 Jul 2003