Feedback on YetAnotherDBCacheContrib

-- ThomasWeigert - 02 Oct 2006

Thanks Thomas for sharing this enhanced contrib. Crawford/Thomas, please consider merging DBCacheContrib and YetAnotherDBCacheContrib soon.

-- PeterThoeny - 03 Oct 2006

Hi Thomas, today I tried to install your Contrib but got following error:

YetAnotherDBCacheContrib_installer.pl: Can't locate object method "new" via package 
"TWiki::Plugins::DBCachePlugin::WebDB" at lib/TWiki/Plugins/DBCachePlugin/Core.pm

Do you have a hint for me please ?

-- ThomasFreudenberg - 05 Nov 2006

Thomas, I have never had good experience with the installer. It is something that is generated in the build process but in my experience does not seem to really work. I never use it.

Can you try to follow the manual process, just install all the dependencies, unzip the application, set the permissions, and then run the test cases? Please advise how this is going.

-- ThomasWeigert - 06 Nov 2006

Hi Thomas,

first of all sorry for late response. I have tested some different installation options and now I have the following infos for you.

1.) After (manual) installation of YetAnotherDBCacheContrib the BlogPlugin does not work anymore. See error info in attached file

2.) After (manual) installation of YetAnotherFormQueryPlugin there ist also an error when try to open FormQueryPlugin. The browser will show:

Can't locate object method "new" via package "TWiki::Plugins::FormQueryPlugin::WebDB" 
at /var/web/twixi.local/htdocs/lib/TWiki/Plugins/FormQueryPlugin/WebDB.pm line 31.

I put in an attachment with all the messages from warning log and also the list of installed plugins in my local wiki. Hope you could fix the problem, maybe there is some dependency missing.

By the way, what is exactliy the difference in DBCacheContrib/!YetAnotherDBCacheContrib and FormQueryPlugin/!YetAnotherFormQueryPlugin ? Thanx for your answer.

-- ThomasFreudenberg - 10 Nov 2006

Thomas, regarding your question (1.), I fear that you cannot use YetAnotherDBCacheContrib if you want to use BlogPlugin. The latter relies on another Plugin, DBCachePlugin, which works only with DBCacheContrib. So I am not surprised that BlogPlugin is affected.

I wonder whether the problem described in (2.) is also related to that you have DBCachePlugin installed. You cannot use these plugins together.

To answer your "by the way" question, the following is included from FormQueryPluginDev to highlight some of the differences...

• In DBCacheContrib I have made changes that hopefully are non-controversial. In detail I
  • made some fixes
  • replaced the parser with a table-driven parser in order to have more flexibility
  • support summing of subfields
  • handle subfield searches
  • allow expansion of fields on rhs of query expression
  • add caseinsensitivity as option
  • add web to cache for topics
  • integrate the Attribute parser from DakarRelease
  • interpret bare string as text=~'string' query
  • allow use of SpreadSheetPlugin in search expression
  • deleted the time-specific operations in favor of using SpreadSheetPlugin to convert dates to numeric format
• In FormQueryPlugin the situation is not as clear. By now, this is almost a different plugin with much functionality and code borrowed from your original work. My main goal was to get as close to %SEARCH% as possible to be able to replace any %SEARCH% by a form query. In detail, I
  • added support for searches over multiple webs
  • support %MATCHCOUNT% as suggested above
  • allow SpreadSheetPlugin computation to be applied to the result of a query as discussed above
  • made "moan" a preference rather than a per call option
  • removed %TOPICCREATOR%
  • removed %ARITHMETIC% (as you can use the SpreadSheetPlugin)
  • removed the color map feature
  • extended %FQPINFO% to support showing results of queries
  • integrate the Attribute parser from DakarRelease
  • add caseinsensitivity as option
  • handle one line at a time so that query memory can be reused in another query
  • added the special variables supported by FormattedSearch
  • changed some of the option names to be more consistent with TWikiSearch (e.g., row_count)
  • numerous fixes
  • more flexibility for using tables
  • add a %DOANDSHOWQUERY% tag
  • support embedding of %SEARCH% and %FORMQUERY% in the format option to allow searches and queries to be applied to the result of a query (a poor man's intersection)

-- ThomasWeigert - 11 Nov 2006

Hi Thomas,

first of its a pity that I could not use the YetAnother... Plugins. From your answer above, I ask myself, why the both Plugins won't come together. It seemed to me, that you have made some good optimization to the original ones. Is it also faster ? As I am using BlogPlugin, do you have an idea about how much work would it be to make it compatible ? The fix has to be done in DBCachePlugin, is this correct ? Greeting, Tom.

-- ThomasFreudenberg - 11 Nov 2006

I have recently performed an experiment replacing the topic-based lookup in XpTrackerPlugin by relying on DBCacheContrib. For those interested, I move the discussion to here...

Test results

Summary

As the table shows, the results do not uniformly favor one or the other approach. Some highlights are

• A topic that does not involve either cache (Main.WebHome), can nevertheless be affected due to plugin handlers being invoked. The DBCacheContrib has no impact, while the xpcache adds a slight performance hit.
• For topics where much analysis is done and many topics have to be read in order to generate the results, the DBCacheContrib is a clear winner, generally by a large margin.
• For topics that do not involve additional topic reads, the need to read the rather large cache file (2,673,246 bytes) for DBCacheContrib imposes a performance penalty of roughly 30%.

Project characteristics

The tests were performed over a reasonably sized data base. The web contained

The sample topics selected had the following characteristics:

The cache loadtime is a critical factor. When that dominates the computation time (or the loadtime of individual topics, DBCacheContrib looses.

Here is a more detailed comparison, for the situation where DBCacheContrib looses, using and adaptation of Crawford's TWikiBench...

Without DBCacheContrib:

With DBCacheContrib:

As you can see, the computation time is obviously better in DBCacheContrib, but the cache load time is 10x as long as without. (There is a small specialized cache in XpTrackerPlugin, which stores relationships between topics, but not the topics themselves. The cache read time in the version using DBCacheContrib is both the loading of the cache and the construction of the in-memory version of the topic relationship cache.)

-- ThomasWeigert - 03 Dec 2006

There is something puzzling here, however. In the timing using ab above, the topic InternalToolsOverheadIteration shows a clear loss for the DBCacheContrib, but when instrumenting the run we find that the difference is not quite as large...

without DBCacheContrib:

with DBCacheContrib:

-- ThomasWeigert - 03 Dec 2006

Anyway, from the detailed timing below on InternalToolsOverheadIteration it appears clear that when the cache is large, but the amount of topic reads required to generate the topic is not large, the cache approach cannot catch up:

without DBCacheContrib:

with DBCacheContrib:

-- ThomasWeigert - 03 Dec 2006

Here for comparison the developer topic where DBCacheContrib wins huge:

without DBCacheContrib:

with DBCacheContrib:

-- ThomasWeigert - 04 Dec 2006

Uploaded support for "managed caches", such that the application can control when to reload the database (obviously this can be much faster, if the application is smart about this).

-- ThomasWeigert - 04 Dec 2006

Bugs:Item4758 YetAnotherDBCacheContrib fails to create a subweb cache

Workaround Create for each web a folder in pub/_work_areas/DBCacheContrib/ with the appropriate owner/group (e.g. www-data).

-- FrankSpangenberg - 02 Oct 2007

Thomas, when you updated YetAnotherDBCacheContrib, the WYSIWYG editor changed a whole bunch of things. Did you change anything besides the TopicClassification from PluginPackage to ContribPackage?

-- PeterThoeny - 2009-07-12