create new tag
, view all tags

Old development discussion for FormQueryPlugin

Refactored by ThomasWeigert, 01 Jun 2005

I've encountered a rather interesting anomaly when this plugin is loaded -- nearly all tables (I say nearly as I have not looked at them all!) that are built using %SEARCH with the embedded format= parameter are incredibly munged up via this plugin. Simple hardcoded tables are not effected. Simply putting the plugin in the disabled list returns order to the "dynamically built" tables.

Crawford, since this is happening on our (Motorola) own internal TWiki server I can send you example topics -- I raise the issue here in case anyone else has run into this same problem. I glanced at the code but it was not obvious what ForQueryPlugin was stepping on.

-- SteveRJones - 10 Sep 2003

I was wondering about something along the lines of this Plugin, but didn't know it existing. Some fairly amazing capabilities. I hope I'll have some time to try it out.

-- JohnTalintyre - 27 Sep 2003

Unfortunately, Crawford has left Motorola and to be quite honest, the plugin needs additional work but no one internally is picking up development. I exchanged some emails with Crawford after his departure and he was not too sure about continuing in the software development field. And I am afraid that I won't be much help.

-- SteveRJones - 06 Dec 2003

Crawford, I will try and send you an example of what is going on - but basically the final vertical bar of a TML spec'd table is somehow removed just by installing FormQueryPlugin and having it initialize. This effects every table in the entire twiki site.

But I am glad you are back ;->

-- SteveRJones - 04 Feb 2004

On reflection, an example isn't going to help. I don't see this behaviour in any other installation, and I can't guess what might be causing it, unless it's some obscure interaction with another plugin. The attached zip contains the very latest version, which I was about to release anyway. Give it a try (read the installation instructions in the plugin topic! There's now a selective enable)

-- CrawfordCurrie - 05 Feb 2004

There is a 'feature' in FormQueryPlugin which conflicts with %SEARCH and TablePlugin; see TablePluginDev. When using %SEARCH to dynamically create a table with a header then FormQueryPlugin joins the generated header table line and first result table line using its commonTagsHandler () subroutine in FormQueryPlugin.pm. The following line is the culprit:

  chop( $text ); # remove trailing NL

This removes the NL on the header table line created by %SEARCH, thus joining it to the first result line, thus messing up the sort by table header feature of the TablePlugin.

Commenting out the above line appears to cure all problems...

-- SimonHardyFrancis - 10 Feb 2004

Crawford, the bug that Simon points out above is precisely the problem that I was seeing on the Austin TWiki server, thus causing me to keep from installing FormQueryPlugin.

-- SteveRJones - 10 Feb 2004

Yeah, I realised that, and that's why I posted the fix here for you to test.

- C

Sorry for fixing a bug that has already been fixed smile I must have downloaded the plugin before the 4th Feb. and somehow overlooked the bug-fix release in the meantime.

-- SimonHardyFrancis - 11 Feb 2004

Yet another question about %TABLEFORMAT:

FormQueryPlugin says '... Any fields in the form in the topic can be output by putting a $ sign before the name of the field. ...' From this then I understand that if I have a form field called FieldOne which is set to the value ValueOne, then I can write into the format part of the statement $FieldOne in order to have the value ValueOne appear in the output. Right? Unfortunately FormQueryPlugin always complains that it doesn't know the field called FieldOne.

Summary: The '$field' mechanism for format output works fine for field names in tables but not for field names in forms. Should this work or have I misunderstood the docs?

-- SimonHardyFrancis - 11 Feb 2004

Hmm, it should. On the assumption you have a search that is returning a topic that contains a form that contains the field "FieldOne", that is. I just checked it and it works fine on form field names for me. Note that the form field has to exist both in the topic and in the current version of the form template, and should be typographically identical. And obey the rules set out for form field naming, of course....

Just a mo - a thought strikes me. Do you have an "extract" in the FORMQUERY? If you do, you are looking at the record for the extracted table and not the topic.

-- CrawfordCurrie - 11 Feb 2004

It's about time you two got out of edit mode and let someone else do some typing (that's my sarcasm directed at Crawford ;-> )

Seriously: As I recall I tried downloading your v1.2 and kept getting timeouts - then I went onto something else and completely forgot that I had tried the download. Hence, I forgot to tell you that I hadn't had a chance to test v1.2. I am now happy to say that I have installed v1.2 and it no longer conflicts with tables created via %SEARCH%!

Now, if I can only figure out what this plugin actually does! <-- (note: sarcasm with a touch of self deprecating humor)

-- SteveRJones - 11 Feb 2004

RTFM, Senor Jones! (damn, I wish they'd add that sarcasm smilie I asked for!). If you want a guided tour, ask HenningSpruth

-- CrawfordCurrie - 11 Feb 2004

You're the plugin-meister! Ok, I added it to the zip - now get Peter to add it to twiki.org.

-- SteveRJones - 11 Feb 2004

Yes I do have an "extract" in the FORMQUERY. So I'm looking at the record for the extracted table and not the topic. Is it possible to have a FormQueryPlugin result table with columns containing information from both extracted tables and topic forms? Want I want: If there are matching rows in the extracted table from a particular TWiki page then I want to output topic form values from the same page. How to do this?

I think so; if I remember rightly, the topic gets entered in each table entry as 'topic' (or $topic or something like that) - this is a pointer to the form of the topic, not the topic name, so "topic.field" ought to work (check the code!) -- CrawfordCurrie

-- SimonHardyFrancis - 16 Feb 2004

Is there a way to limit the results in the result table? E.g. Only list result rows from 10 to 19?

None that I've coded. Don't let that stop you.... hehe! -- CrawfordCurrie

-- SimonHardyFrancis - 16 Feb 2004

The default TWiki SEARCH{} format is still broken by FormQueryPlugin, putting the first result on the same line as the header.

-- ArthurClemens - 17 Feb 2004

Arthur, please check /lib/TWiki/Plugins/FormQueryPlugin.pm, function commonTagsHandler() for the following line near the end:

  chop( $text ) if ( $text =~ m/\n$/so ); # remove trailing NL

The bug is only fixed if this line is present. Also, the bug fixed version is only available for download from FormQueryPluginDev and not FormQueryPlugin.

If you're still having problems after checking the above stuff then please post an example of the %SEARCH command which is going wrong on your TWiki installation.

-- SimonHardyFrancis - 18 Feb 2004

Crawford, this looks like a really powerful tool, and just what I need for my company project site. However, what I miss on the FormQueryPlugin page is a description of needed steps to get this running. So I need to set the FQTABLES preference in the WebPreferences topic? Can I do this in TWikiPreferences instead? And the topics that are listed there must be formatted using %EDITTABLE? Indifferent how many columns? Do I need to set header and format? Should I just define the headers, like:

%EDITTABLE{header="|*value1*|* value2*|*value3*|" format="|text, 20|text,20|text,20|"}%
| *Question* | *Answer* | *Projectmanager* |
| How to render the date of found attachments in human readable fashion? %<nop>CALC{"$FORMATGMTIME($date, $day $mon $year)"}% does only show todays's date | I believe that is to do with the plugin rendering order. I've had similar problems, and I think that may be your answer. |  |
| How to render the file size as in the attachment table, with K instead of bytes? I can't use SpreadSheetPlugin inside the format it seems. Earlier comment: change the loading order of plugins. New question: what should the order be? |  |  |
| I want to have a list of attachments, and next to each file link the topic's (where the attachment is in) name, which is a form field. So the question: how to access form fields from a topic when I only have a list of attachments? Noting that _there's no way to do an =INTERSECTION= at present_. |  |  |
| Is there an equivalent to the exclude= function of search? My query is returning the template topic. | I would add "AND topic!=templateName" to the search parameter -- Main.PeterWeatherdon - 05 Jan 2005 |  |
| 1) Is there a way to specify the web? 2) http://wiki.conceptmapping.org/Sandbox/PhotoGallery is trying to show the URLs for all pictures found in the People web. It almost works but the path names break because they end up with spaces in them. Is this a bug? |  |  |
| Strange since it's main value add on top of the regular SEARCH facility is to search embedded tables, so <nop>TableQueryPlugin? Main.TimSidel, Main.MartinCleaver |  |  |
| It would be useful to have a new demo site, perhaps on http://twikiplugins.sf.net/? |  |  |
| I'd welcome DBCacheContrib into the TWikiKernal - perhaps as a TWiki::Data:: |  |  |

-- ArthurClemens - 17 Feb 2004

So if the table needs to be a WebForm, how does a TaskTable then look like? Do I need to use the | Name | Type | Size | Values | Tooltip message | format?

A working example would be greatly appreciated!

-- ArthurClemens - 18 Feb 2004


Arthur, I'll trade you a full set of user documentation for a public release of your http://www.visiblearea.com/cgi-bin/twiki/view/Patterns/Home_ skin as the TWiki default skin hehe! (friendly)

Simon, the OWiki people offered to host a demo; I provided them with a zip of my own canned demo. See http://chaos.owiki.org/FQP/ - I don't know if it works yet.

-- CrawfordCurrie - 18 Feb 2004

Nice plugin - was thinking about implementing something, then found this plugin.

I've been experiencing some oddities when using this with mod_perl. For example, different threads get different versions of the same macro, so that refreshing the page several times produce different results each time. What's the best approach here?

One thing that's not clear with TWiki is which plugins are supported within mod_perl.

-- TimSlidel - 29 Mar 2004

Hmmm. I hesitate to say it, Tim, but how does "don't use mod_perl" sound to you?

I did try experimenting with mod_perl, but I gave up after a while, because I kept seeing non-deterministic behaviours. I'd love to blame the plugins discovery mechanism TWiki uses, but the truth is, I really don't know why.

Good point about mod_perl support. The way plugins are discovered by twiki makes it quite unlikely IMHO that any plugin gains benefit from mod_perl. In truth, I don't completely trust the sopport for the main twiki under mod_perl, I wouldn't swear to it. But no plugin should break under mod_perl.

-- CrawfordCurrie - 29 Mar 2004

Weeelll, not so good really. I've been running TWiki for a year with mod_perl and it's been generally good. I also noted the comments below (from ADatabaseCalledTWiki) so assumed that this plugin would be ok under mod_perl. I'd miss the great performance gain that mod_perl gives for TWiki in general and it looks like from your comments that this would apply to FormQueryPlugin too?

BTW I did some benchmarks on a 2GHz lightly loaded linux/apache server with mod_perl and Storable. Loading a 2000 topic web takes ~0.6s reading topics and ~0.3s from the cache, and 1.2s to perform a complex query that visits every topic and every embedded table. Call this 1.8/1.5. Lose Storable, I get 2.8/2.2. Lose mod_perl, 5.0/4.8.

I'm rerunning the benchmarks on a heavily loaded server; I'll report those later.

-- CrawfordCurrie - 30 Jun 2003

As I understand it then FormQueryPlugin first searches for all pages with WebForm fields matching a particular criteria. Then as a secondary step FormQueryPlugin can 'extract' all tables (of a particular type as declared by the 'include' parameter of the %EDITTABLE statement) from those found pages.

I have not attempted to get FormQueryPlugin to do a search without using a WebForm on each page where a TWiki table can also be found. Just by looking at the syntax of the FormQueryPlugin statements then I doubt if this would work - because probably 'extract' always needs to be used in conjunction with 'search'.

I have never managed to get FormQueryPlugin to include a form field in my result table (although the special case $topic works). I haven't looked into the source code about this but my guess is that as soon as you 'extract' tables then the current query enters a different context and the possibility of rendering form fields disappears. However I'm not sure if this is intended behavior or even if I understood the docs about this feature properly smile Currently I use FormQueryPlugin exclusively for generating dynamic TWiki tables from queries on static TWiki tables.

-- SimonHardyFrancis - 18 Feb 2004

I think Storable has locking so I would guess that multiple plugin server threads can't overwrite the cache, which would be the main problem. I'm not entirely sure what happens about the macros and why they might be cached differently. Clearly once you've got a working macro you don't tend to change it that much, so I'm not sure that the problem I experienced is a show stopper, esp if you restart the web server to clear out any caches.

Can you shed any light on what complications mod_perl might add for this plugin?

-- TimSlidel - 31 Mar 2004

I think this does what I want: add all rows from several same-type tables in a topic. This is useful when, for example, table tasks are interspersed with relevant comments or analysis in a single topic. Previous behaviour is to take the last table in the topic. Can't see a drawback in allowing all tables.

--- WebDB.pm    Wed Mar 31 14:35:47 2004
+++ WebDB.pm.new        Wed Mar 31 14:34:41 2004
@@ -158,7 +158,10 @@
        my $tablename = $1;
        my $ttype = $tables{$tablename};
        if ( defined( $ttype )) {
-         my $table = new FormQueryPlugin::Array();
+          my $table = $meta->fastget( $tablename );
+          if ( !defined( $table )) {
+             $table = new FormQueryPlugin::Array();
+          }
          my $lc = 0;
          my $row = "";
          while ( $line = <FH> ) {

-- TimSlidel - 31 Mar 2004

Another patch I like is to forward all GET parameters from the autocreate script so that I can set form values for new autocreated topics by using an HTML form directly instead of TOPICCREATOR, e.g.

<form action="%SCRIPTURLPATH%/autocreate/%INTURLENCODE{"%WEB%"}%/PrdNo1">
   <input type="hidden" name="topicparent" value="%TOPIC%" />
   <input type="hidden" name="relation" value="copy" />
   <input type="hidden" name="templatetopic" value="ProductTemplate" />
   <input type="hidden" name="formtemplate" value="ProductForm" />
   <input type="hidden" name="onlywikiname" value="on" />
   <input type="text" name="Name" value="Enter name" size="15"/>
   <input type="text" name="Description" value="Enter description" size="50"/>
   <input type="submit" value="Add new product" />


--- autocreate  Wed Mar 31 14:47:23 2004
+++ autocreate.new      Wed Mar 31 14:49:21 2004
@@ -38,12 +38,7 @@
 $nt =~ s/\n/$max/o;

-my $dir = TWiki::Func::getScriptUrl( $webName, $nt, "edit")."?";
-my $p = $query->param( "formtemplate" );
-$dir .= "formtemplate=$p;" if ( $p );
-$p = $query->param( "templatetopic" );
-$dir .= "templatetopic=$p;" if ( $p );
+my $dir = TWiki::Func::getScriptUrl( $webName, $nt, "edit")."?".$query->query_string();
 print $query->redirect( $dir );


Note that the patch will only work for GET forms and will fail for POST. TOPICCREATOR uses GET and form-writers probably will too, so I think this should be ok. Can't see an easy way to support generic POSTs here.

-- TimSlidel - 31 Mar 2004

A little modification so that macros work in mod_perl. In mod_perl the macros hash lives through CGI script calls (so the macros are effectively cached). This patch adds a hash of FileTime objects that gets checked to see if a macro needs to be reloaded (because it was modified since the last load). This resolves the problems with macros not being reloaded reliably under mod_perl.

--- ReqDBSupport.pm     Sun Jul 20 14:16:10 2003
+++ ReqDBSupport.pm.new Wed Mar 31 18:18:50 2004
@@ -14,6 +14,7 @@
 { package FormQueryPlugin::ReqDBSupport;

   my %macros;
+  my %macro_times;

   # Expand a macro. The macro is identified by the 'topic' parameter
@@ -23,14 +24,17 @@
   sub callMacro {
     my ( $macro, $params, $web, $topic ) = @_;
     my $attrs = new FormQueryPlugin::Map( $params );
+    my $dataDir = TWiki::Func::getDataDir() . "/$web";

     my $mtop = $attrs->fastget( "topic" );
+    my $filename = "$dataDir/$mtop.txt";

-    if ( !defined( $macros{$mtop} )) {
+    if ( !defined( $macros{$mtop} ) || !$macro_times{$mtop}->uptodate() ) {
       if ( !TWiki::Func::topicExists( $web, $mtop )) {
        return FormQueryPlugin::WebDB::moan( $macro, $params, "No such macro $mt
op ", "" );
       my $text = TWiki::Func::readTopicText( $web, $mtop );
+      $macro_times{$mtop} = new FormQueryPlugin::FileTime( $filename );
       $text =~ s/%META:\w+{.*?}%//go;
       $macros{$mtop} = $text;

-- TimSlidel - 02 Apr 2004

Tim, I'm looking at your patches now. I assume you re-ran all the tests with the patches in place?

-- CrawfordCurrie - 02 Apr 2004

OK, I incorporated all your patches, they all look OK. All tests pass, though I haven't written tests for the functionality in the patches because those really should have been provided with the patches >:-). I haven't updated the release yet; instead, get the latest revision from anonymous CVS or the zip below.

-- CrawfordCurrie - 02 Apr 2004

Was away for Easter. Thanks for incorporating the patches. I hate to say that I hadn't run the tests, but agree that I should have and that I should have provided test revisions for the modifications. I'll try to do those things at some point, esp if I do more patching.

-- TimSlidel - 15 Apr 2004

I have been playing with this plugin trying to solve my problems (see ActionTrackerDoesntExpandAttributes) and I thought I would add my .02 worth.

First I have a minor bug report. This was in the zip file from FormQueryPlugin. There is a function in FileTime.PM Line 23 called uptodate. This is referred to by WebDB.pm on Line 316 correctly. In ReqDBSupport.pm on Line 33 this is called as uptodat and needs the e added.

!$macro_times{$mtop}->uptodat() ) { # TimSlidel 1/4/04
should be </br/> !$macro_times{$mtop}->uptodate() ) { # TimSlidel 1/4/04

This caused an error when saving a topic with more than one CALLMACRO statement.
I also had some wierd behavior where SET didn't work unless there was an ARITH command also in the topic, and this also went away after the fix above although I have no idea if it was related because it didn't generate an error.

OK, enough whining from me. Overall I am very impressed with this plugin, particularly the SET, CALLMACRO, and TOPICCREATOR functions.

-- JeffEaston - 25 Jun 2004

Thanks for the bugfix!

I too recently encountered the need for nested CALLMACRO, so I just coded it. No protection against infinite recursion, though, so be warned. Not checked in yet, so don't expect it to suddenly appear. I'm going to split up the plugin as per the discussion with Thomas first.

I find it hard to imagine a SEARCH with more than one line. Can you give me an example of why you need multi-line SET?

-- CrawfordCurrie - 29 Jun 2004


I'm starting to use the FormQueryPlugin. It's very very good! But, the long documentation is not done and it have a low number of examples. frown

Now i'm tring to use the attachments array to do a query and to show on finded result. I don't know how to use the attachments array to do a query, but I'm on the way to use it to show on result. ... But, I don't know how to use it... I put %TABLEFORMAT{ name=Test format=" $attachments " }% and I recive FormQueryPlugin::Array=HASH(0x8974e98)

The question is: How I use the attachments with the FormQueryPlugin?

-- AurelioAHeckert - 07 Jul 2004

Aurelio, I have documented attachments in the release doc for the DBCachePlugin, which is going to be one of three plugins that the FormQueryPlugin is being split into (the other one is MacrosPlugin, already released). I hope to have the split done today; all the code is written, all I have to do is make sure all the tests pass. Note that at this stage I'm testing against the Cairo "bleeding edge" i.e. the code in subversion, and can't make any guarantees about earlier TWiki code (though it should all work with Beijing).

-- CrawfordCurrie - 08 Jul 2004

There is a rather interesting line in FormQueryPlugin that is broken off:

Actiually, this isn't quite enough. What we really want to do is list the

I am curious how this story ends smile

-- ArthurClemens - 21 Sep 2004

I can't figure it out by myself. I would like to create a table of attachments of the past week, with direct links to the files. If I use $_up.topic$path, the topic name is rendered as link, so I can't use it with %PUBURL% etc.

Also one of my filenames gets rendered as wikilink. So I would like to nop the filename, but how?

-- ArthurClemens - 21 Sep 2004

Okay, strange problem. I can get a query to work in Preview mode, but after Saving the topic all I get are the unrendered variables. Why is the topic rendered correctly only in Preview mode?

-- BillGunter - 25 Feb 2005

Hello guys, do you know why the SUMFIELD feature is not working?, we upgraded to the latest plugin of FormQueryPlugin and we see that the SUMFILED returns always 0.

-- GustavoAdolfoLopez - 21 Mar 2005

Edit | Attach | Watch | Print version | History: r3 < r2 < r1 | Backlinks | Raw View | Raw edit | More topic actions
Topic revision: r3 - 2005-06-01 - ThomasWeigert
  • Learn about TWiki  
  • Download TWiki
This site is powered by the TWiki collaboration platform Powered by Perl Hosted by OICcam.com Ideas, requests, problems regarding TWiki? Send feedback. Ask community in the support forum.
Copyright © 1999-2017 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.