Development discussion on DBCachePlugin

initial release.

-- MichaelDaum - 24 Nov 2005

Michael, wow, yet another Plugin! Thanks for sharing with us

-- PeterThoeny - 24 Nov 2005

Michael, does this interface with the original DBCacheContrib, or with that contrib as I had modified it to include a better parser and support all the extensions made to FormQueryPlugin to make it more consistent with Search and allow integration with the SpreadSheetPlugin?

-- ThomasWeigert - 03 Dec 2005

The DBCachePlugin's primary purpose is to replace INCLUDE and SEARCH with DBCALL and DBQUERY. Basically, it is a lightweighted frontend to the DBCacheContrib, "lightweighted" in the sense that I left out some of the features in the DBCC.

-- MichaelDaum - 06 Dec 2005

Hi,

I've added a bit of magic from BlogPlugin to parse 'Date' form fields; I needed it for my KnowledgeBase TWikiApplication. I've attached the (unified) diff here.

-- EricCote - 12 Mar 2006

Michael, sorry for talking a while to come back to this. Can you be expand on whether the DBCC you interface with is the original DBCC or the DBCC with the many changes that I made to improve search, spreadsheet integration, etc.?

-- ThomasWeigert - 15 Jun 2006

Ehm, I just took the stuff that was there and created some interface around it. So I assume the answer to your question is "yes" though I am not exactly sure what you mean, i.e. spreadsheet integration.

-- MichaelDaum - 15 Jun 2006

What I meant is the following: I had generated a large number of updates to the original DBCC which were attached to the Dev topic. These consisted of a more flexible parser, a more consistent treatment of search so that DBCC could replace standard %SEARCH% almost transparently, and finally an interface that allows the spreadsheet plugin to perform computations over results of the DBCC. I was wondering whether the wrapper you provided was developed against the original DBCC (the one attached to the DBCC topic) or against the updated DBCC (the one attached to the dev topic).

-- ThomasWeigert - 16 Jun 2006

Oh dear. How could I have been aware of your nice work lingering around? Replacing SEARCH transparently using DBCC would be GREAT news for TWiki. You should have SHOUTED OUT LOUDLY and pushed forward so that your patch would have been integrated into the official release.

Now, the situation is different: the DBCP is a rather light-weighted frontend to DBCC, offering much more flexibility for TWikiApplications than SEARCH. There are also features in there that aren't available somewhere else (DBSTATS); it works around some of the rough edges / missing features of DBCC etc.

So how can we resolve this now? Having a DBCacheSearchPlugin that provides a new SEARCH implementation based on DBCC makes totally sense. I never had this demand on my paper doing DBQUERY though migrating SEARCH to DBQUERY should be fairly easy.

The question for me is: how invasive is your DBCC patch? Does it still apply w/o errors? Does it break DBCP and the BlogPlugin? Can you outline what extra-features you added to the (search string?) parser, please. Or did you already somewhere else? Maybe it is worth porting DBCP to your version of the DBCC and work onwards from there. This is a bit of work but definitely worth it.

-- MichaelDaum - 16 Jun 2006

OK. If we both are willing to put some effort in it should be no big deal. I'll start investigating what needs to be done to rev the DBCC changes to Dakar. I'll document the questions you have above, but if you could create a table of answers you need, I'll populate it, to get started.

-- ThomasWeigert - 22 Jun 2006

When I enable this plugin, I get the following error when running the installation scripts of other plugins:

ERROR: bad Map expression at -ContractAudit at lib/TWiki/Contrib/DBCacheContrib/Map.pm line 125.

With DBCachePlugin disabled, the installation script works fine. The script I used to recreate this was UserInfoPlugin _installer.pl, although I remember getting this message with all other installation scripts I used when doing our recent updgrade. I am currently using TWiki 04. Any ideas?

Update: When I try and save topics in the Main web, I get the following error when DBCachePlugin is enabled:

ERROR: bad Map expression at -ContractAudit at /usr/lib/perl5/5.8.5/CGI/Carp.pm line 314. at /usr/lib/perl5/5.8.5/CGI/Carp.pm line 314

There was more to that error but I left it out. With the plugin disabled, everything is fine. Also, this message was shown in Firefox, but in IE all we get is a 'Server Not Available error'. Im sure I can't be the only person who gets this!

Moved this question to DBCachePluginProblems

-- AndrewRJones - 31 Aug 2006

I had to make a few small modifications.

First, the plugin incorrectly handles '0' fields. This value is implicitly converted to false, so the || '' and related tests were dropping real data (and causing a CALC to fail) This is the bulk of the change. (Actually, a large part is some renaming I did to make sense of what happening.)

Second, which probably needs more testing, is a one line addition to DBQUERY - when I table formatted the results, extra lines were being inserted and destroying the table. It turns out that SpreadSheetPlugin does a split on newline, and then appends a newline afterwards - since DBCachePlugin is passing unterminated single lines to it, extra newlines get appended.

Making the indentation consistant (some is tabs and some is spaces) might be nice as well, but it would have added a lot of noise to the diff.

diff original/Core.pm modified/Core.pm
132a133
>       $format =~ s/\n$//; #Remove the trailing \n that may have been added by CALC
425c426
<     $result .= "<tr><th>$key</th><td>$value</td></tr>\n" if $value;
---
>     $result .= "<tr><th>$key</th><td>$value</td></tr>\n" if defined($value);
437c438
<       $result .= "<tr><th>$key</th><td>$value</td>\n" if $value;
---
>       $result .= "<tr><th>$key</th><td>$value</td>\n" if defined($value);
diff original/WebDB.pm modified/WebDB.pm
142,144d141
<   
<   my $form = $topicObj->fastget('form');
<   return '' unless $form;
146,147c143,151
<   $form = $topicObj->fastget($form);
<   my $formfield = $form->fastget($theFormField) || '';
---
>   my $formName = $topicObj->fastget('form');
>   return '' unless $formName;
> 
>   my $formObj = $topicObj->fastget($formName);
>   my $formfield = $formObj->fastget($theFormField);
>   if (!defined($formfield))
>   {
>     $formfield = '';
>   }

-- JustinLove - 05 Jan 2007

FYI, the latest version of the SpreadSheetPlugin (18 Dec 2006, r12315) does no longer append a newline at the end.

-- PeterThoeny - 05 Jan 2007

The problem whereby DBCachePlugin throws up an error message when a topic is saved if any topic in that web has mal-formed metadata is being addressed with Bugs:Item4967. The error will be logged in the apache error log but the user won't see it. Michael uploaded a new version yesterday that incorporates this fix.

-- LynnwoodBrown - 15 Nov 2007

I get a "bad map expression" error when I create a BlogUp web using the BlogFactory function of BlogPlugin, here is my apache.log

 [error]  ERROR: bad Map expression at /var/www/twiki/lib/TWiki/Contrib/DBCacheContrib/Map.pm line 126, referer: http://hostname/twiki/bin/view/TWiki/BlogFactory
 [error]  \tTWiki::Contrib::DBCacheContrib::Map::get('TWiki::Plugins::DBCachePlugin::WebDB=HASH(0x8c14654)', ' ') called at /var/www/twiki/lib/TWiki/Contrib/DBCacheContrib.pm line 244, referer: http://hostname/twiki/bin/view/TWiki/BlogFactory
 [error]  \tTWiki::Contrib::DBCacheContrib::_onReload('TWiki::Plugins::DBCachePlugin::WebDB=HASH(0x8c14654)', 'ARRAY(0x889e5f4)') called at /var/www/twiki/lib/TWiki/Contrib/DBCacheContrib.pm line 340, referer: http://hostname/twiki/bin/view/TWiki/BlogFactory
 [error]  \tTWiki::Contrib::DBCacheContrib::load('TWiki::Plugins::DBCachePlugin::WebDB=HASH(0x8c14654)') called at /var/www/twiki/lib/TWiki/Plugins/DBCachePlugin/WebDB.pm line 47, referer: http://hostname/twiki/bin/view/TWiki/BlogFactory
 [error]  \tTWiki::Plugins::DBCachePlugin::WebDB::load('TWiki::Plugins::DBCachePlugin::WebDB=HASH(0x8c14654)') called at /var/www/twiki/lib/TWiki/Plugins/DBCachePlugin/Core.pm line 928, referer: http://hostname/twiki/bin/view/TWiki/BlogFactory
 [error]  \tTWiki::Plugins::DBCachePlugin::Core::getDB('Main') called at /var/www/twiki/lib/TWiki/Plugins/DBCachePlugin/Core.pm line 59, referer: http://hostname/twiki/bin/view/TWiki/BlogFactory
 [error]  \tTWiki::Plugins::DBCachePlugin::Core::afterSaveHandler('%META:TOPICINFO{author="AllanEspinosa" date="1202660194" form...', 'WebPreferences', 'Blog', '', 'TWiki::Meta=HASH(0x84e2940)') called at /var/www/twiki/lib/TWiki/Plugins/DBCachePlugin.pm line 91, referer: http://hostname/twiki/bin/view/TWiki/BlogFactory

-- AllanEspinosa - 10 Feb 2008

Is it possible to disable DBCachePlugin on a per-web basis? We have one large web with over 12,150 topics, and frequently saving any topic in that web started taking over three minutes to complete. Once I disabled DBCachePlugin for our entire installation, saves went back to completing in a second or two.

Any suggestions for working around this would be greatly appreciated.

Many thanks, Devin

-- DevinBougie - 24 Feb 2009

The Plugins API does not support selective disabling per web. You can however hack the plugin's initPlugin subroutine to return 0 if you are in the big web.

-- PeterThoeny - 25 Feb 2009