Tags:
create new tag
, view all tags

YetAnotherDBCacheContribDev Discussion: Page for developer collaboration, enhancement requests, patches and improved versions on YetAnotherDBCacheContrib contributed by the TWikiCommunity.
• Please let us know what you think of this extension.
• For support, check the existing questions, or ask a new support question in the Support web!
• Please report bugs below

Feedback on YetAnotherDBCacheContrib

-- ThomasWeigert - 02 Oct 2006

Thanks Thomas for sharing this enhanced contrib. Crawford/Thomas, please consider merging DBCacheContrib and YetAnotherDBCacheContrib soon.

-- PeterThoeny - 03 Oct 2006

Hi Thomas, today I tried to install your Contrib but got following error:

YetAnotherDBCacheContrib_installer.pl: Can't locate object method "new" via package 
"TWiki::Plugins::DBCachePlugin::WebDB" at lib/TWiki/Plugins/DBCachePlugin/Core.pm

Do you have a hint for me please ?

-- ThomasFreudenberg - 05 Nov 2006

Thomas, I have never had good experience with the installer. It is something that is generated in the build process but in my experience does not seem to really work. I never use it.

Can you try to follow the manual process, just install all the dependencies, unzip the application, set the permissions, and then run the test cases? Please advise how this is going.

-- ThomasWeigert - 06 Nov 2006

Hi Thomas,

first of all sorry for late response. I have tested some different installation options and now I have the following infos for you.

1.) After (manual) installation of YetAnotherDBCacheContrib the BlogPlugin does not work anymore. See error info in attached file

2.) After (manual) installation of YetAnotherFormQueryPlugin there ist also an error when try to open FormQueryPlugin. The browser will show:

Can't locate object method "new" via package "TWiki::Plugins::FormQueryPlugin::WebDB" 
at /var/web/twixi.local/htdocs/lib/TWiki/Plugins/FormQueryPlugin/WebDB.pm line 31.

I put in an attachment with all the messages from warning log and also the list of installed plugins in my local wiki. Hope you could fix the problem, maybe there is some dependency missing.

By the way, what is exactliy the difference in DBCacheContrib/!YetAnotherDBCacheContrib and FormQueryPlugin/!YetAnotherFormQueryPlugin ? Thanx for your answer.

-- ThomasFreudenberg - 10 Nov 2006

Thomas, regarding your question (1.), I fear that you cannot use YetAnotherDBCacheContrib if you want to use BlogPlugin. The latter relies on another Plugin, DBCachePlugin, which works only with DBCacheContrib. So I am not surprised that BlogPlugin is affected.

I wonder whether the problem described in (2.) is also related to that you have DBCachePlugin installed. You cannot use these plugins together.

To answer your "by the way" question, the following is included from FormQueryPluginDev to highlight some of the differences...

  • In DBCacheContrib I have made changes that hopefully are non-controversial. In detail I
    • made some fixes
    • replaced the parser with a table-driven parser in order to have more flexibility
    • support summing of subfields
    • handle subfield searches
    • allow expansion of fields on rhs of query expression
    • add caseinsensitivity as option
    • add web to cache for topics
    • integrate the Attribute parser from DakarRelease
    • interpret bare string as text=~'string' query
    • allow use of SpreadSheetPlugin in search expression
    • deleted the time-specific operations in favor of using SpreadSheetPlugin to convert dates to numeric format
  • In FormQueryPlugin the situation is not as clear. By now, this is almost a different plugin with much functionality and code borrowed from your original work. My main goal was to get as close to %SEARCH% as possible to be able to replace any %SEARCH% by a form query. In detail, I
    • added support for searches over multiple webs
    • support %MATCHCOUNT% as suggested above
    • allow SpreadSheetPlugin computation to be applied to the result of a query as discussed above
    • made "moan" a preference rather than a per call option
    • removed %TOPICCREATOR%
    • removed %ARITHMETIC% (as you can use the SpreadSheetPlugin)
    • removed the color map feature
    • extended %FQPINFO% to support showing results of queries
    • integrate the Attribute parser from DakarRelease
    • add caseinsensitivity as option
    • handle one line at a time so that query memory can be reused in another query
    • added the special variables supported by FormattedSearch
    • changed some of the option names to be more consistent with TWikiSearch (e.g., row_count)
    • numerous fixes
    • more flexibility for using tables
    • add a %DOANDSHOWQUERY% tag
    • support embedding of %SEARCH% and %FORMQUERY% in the format option to allow searches and queries to be applied to the result of a query (a poor man's intersection)

-- ThomasWeigert - 11 Nov 2006

Hi Thomas,

first of its a pity that I could not use the YetAnother... Plugins. From your answer above, I ask myself, why the both Plugins won't come together. It seemed to me, that you have made some good optimization to the original ones. Is it also faster ? As I am using BlogPlugin, do you have an idea about how much work would it be to make it compatible ? The fix has to be done in DBCachePlugin, is this correct ? Greeting, Tom.

-- ThomasFreudenberg - 11 Nov 2006

I have recently performed an experiment replacing the topic-based lookup in XpTrackerPlugin by relying on DBCacheContrib. For those interested, I move the discussion to here...

Test results

Test xpache DBCacheContrib Ratio
req/sec req ms req/sec req ms
SdaTeam 0.08 11999.20 0.16 6178.80 0.515
SdaTeamDetail 0.04 24351.00 0.08 12005.20 0.493
CodeGenServicesProject 0.27 3699.20 0.23 4274.00 1.155
MousetrapSecurityProject 0.29 3474.80 0.24 4204.00 1.21
MtRel025700 0.18 5614.00 0.17 5962.40 1.062
MtRel030100 0.24 4111.80 0.20 5503.80 1.339
InternalToolsOverheadIteration 0.25 3997.60 0.20 5027.20 1.258
ToolMaintenanceDeveloper 0.07 14184.20 0.16 6387.00 0.45
SdaCq5728Story 0.25 4033.80 0.28 3581.00 0.888
TrainingOverhead2006Story 0.29 3483.00 0.29 3503.00 1.006

Summary

As the table shows, the results do not uniformly favor one or the other approach. Some highlights are

  • A topic that does not involve either cache (Main.WebHome), can nevertheless be affected due to plugin handlers being invoked. The DBCacheContrib has no impact, while the xpcache adds a slight performance hit.
  • For topics where much analysis is done and many topics have to be read in order to generate the results, the DBCacheContrib is a clear winner, generally by a large margin.
  • For topics that do not involve additional topic reads, the need to read the rather large cache file (2,673,246 bytes) for DBCacheContrib imposes a performance penalty of roughly 30%.

Project characteristics

The tests were performed over a reasonably sized data base. The web contained

Projects 2
Teams 8
Iterations 111
Stories 1137

The sample topics selected had the following characteristics:

Topic Characteristics
SdaTeam Uses XPSHOWPROJECTTEAMS, XPSHOWPROJECTITERATIONS, XPSHOWLOAD. Involves 15 developers and 30 active iterations.
SdaTeamDetail Uses XPSHOWPROJECTCOMPLETIONBYSTORIES, XPSHOWPROJECTCOMPLETIONBYTASKS, XPSHOWPROJECTSTORIES. Involves 100 total iterations, 1104 stories (219 not started, 70 in progress, and 815 completed), 2421 tasks (472 not started, 70 in progress, and 1879 completed).
CodeGenServicesProject Uses XPSHOWTEAMITERATIONS. Large project, involves 74 Iterations, of which 6 are still in progress.
MousetrapSecurityProject Uses XPSHOWTEAMITERATIONS. Small project, involves 2 Iterations, all of which are still in progress.
MtRel025700 Uses XPSHOWITERATIONTERSE, XPSHOWITERATION, XPVELOCITIES, XPCOQ. Large iteration, involves 86 stories, all completed. 10 developers.
MtRel030100 Medium iteration, involves 19 stories, all in progress. 5 developers.
InternalToolsOverheadIteration Uses XPSHOWITERATIONTERSE, XPSHOWITERATION, XPVELOCITIES, XPCOQ. Small iteration, involves 10 stories, most of them ongoing stories (more tasks per story than average). 11 developers.
ToolMaintenanceDeveloper Uses XPSHOWDEVELOPERTIMESHEET, XPSHOWDEVELOPERESTIMATE, XPSHOWLOAD (for this developer only). 67 rows in timesheet, 62 rows in estimate table, involved in 5 iterations.
SdaCq5728Story Uses XPSHOWTASKTABLE. Normal sized story, 2 tasks.
TrainingOverhead2006Story Uses XPSHOWTASKTABLE. Overhead story, 11 tasks.

The cache loadtime is a critical factor. When that dominates the computation time (or the loadtime of individual topics, DBCacheContrib looses.

Here is a more detailed comparison, for the situation where DBCacheContrib looses, using and adaptation of Crawford's TWikiBench...

Without DBCacheContrib:

Mark Delta Absolute
time usr sys cpu time usr sys cpu
start xpShowIterationTerse 1.24169 0.74 0.39 1.13 1.24169 0.74 0.39 1.13
start cache read 0.000559092 0.00 0.00 0.00 1.24224 0.74 0.39 1.13
finish cache read 0.021462 0.03 0.00 0.03 1.26371 0.77 0.39 1.16
finish xpShowIterationTerse 0.18348 0.15 0.02 0.17 1.44719 0.92 0.41 1.33

With DBCacheContrib:

Mark Delta Absolute
time usr sys cpu time usr sys cpu
start xpShowIterationTerse 1.15176 0.69 0.45 1.14 1.15176 0.69 0.45 1.14
start cache read 0.00058198 0.00 0.00 0.00 1.15234 0.69 0.45 1.14
finish cache read 0.433366 0.37 0.03 0.40 1.58571 1.06 0.48 1.54
finish xpShowIterationTerse 0.048466 0.05 0.00 0.05 1.63417 1.11 0.48 1.59

As you can see, the computation time is obviously better in DBCacheContrib, but the cache load time is 10x as long as without. (There is a small specialized cache in XpTrackerPlugin, which stores relationships between topics, but not the topics themselves. The cache read time in the version using DBCacheContrib is both the loading of the cache and the construction of the in-memory version of the topic relationship cache.)

-- ThomasWeigert - 03 Dec 2006

There is something puzzling here, however. In the timing using ab above, the topic InternalToolsOverheadIteration shows a clear loss for the DBCacheContrib, but when instrumenting the run we find that the difference is not quite as large...

without DBCacheContrib:

Mark Delta Absolute
time usr sys cpu time usr sys cpu
start xp tags 1.22195 0.66 0.46 1.12 1.22195 0.66 0.46 1.12
finish xp tags         1.37356 0.75 0.50 1.25

with DBCacheContrib:

Mark Delta Absolute
time usr sys cpu time usr sys cpu
start xp tags 1.159 0.74 0.40 1.14 1.159 0.74 0.40 1.14
finish xp tags         1.66377 1.12 0.49 1.61

-- ThomasWeigert - 03 Dec 2006

Anyway, from the detailed timing below on InternalToolsOverheadIteration it appears clear that when the cache is large, but the amount of topic reads required to generate the topic is not large, the cache approach cannot catch up:

without DBCacheContrib:

Mark Delta Absolute
time usr sys cpu time usr sys cpu
start xpShowIterationTerse 1.31189 0.72 0.45 1.17 1.31189 0.72 0.45 1.17
start cache read 0.000571966 0.00 0.00 0.00 1.31246 0.72 0.45 1.17
finish cache read 0.01212 0.01 0.00 0.01 1.32458 0.73 0.45 1.18
finish xpShowIterationTerse 0.0208271 0.01 0.01 0.02 1.34541 0.74 0.46 1.20
start xpShowIteration 0.0528951 0.03 0.01 0.04 1.39831 0.77 0.47 1.24
finish xpShowIteration 0.0207529 0.02 0.00 0.02 1.41906 0.79 0.47 1.26
start show pivot 0.00125194 0.00 0.00 0.00 1.42031 0.79 0.47 1.26
finish show pivot 0.020627 0.01 0.00 0.01 1.44094 0.80 0.47 1.27
start show pivot 0.000838995 0.00 0.00 0.00 1.44178 0.80 0.47 1.27
finish show pivot 0.015543 0.02 0.00 0.02 1.45732 0.82 0.47 1.29
Finish 0.505508 0.36 0.05 0.41 1.96283 1.18 0.52 1.70

with DBCacheContrib:

Mark Delta Absolute
time usr sys cpu time usr sys cpu
start xpShowIterationTerse 1.52091 0.71 0.38 1.09 1.52091 0.71 0.38 1.09
start cache read 0.000591993 0.00 0.00 0.00 1.5215 0.71 0.38 1.09
finish cache read 0.510143 0.36 0.08 0.44 2.03165 1.07 0.46 1.53
finish xpShowIterationTerse 0.00727701 0.00 0.00 0.00 2.03892 1.07 0.46 1.53
start xpShowIteration 0.0550821 0.04 0.01 0.05 2.09401 1.11 0.47 1.58
finish xpShowIteration 0.00672483 0.01 0.00 0.01 2.10073 1.12 0.47 1.59
start show pivot 0.000868082 0.00 0.00 0.00 2.1016 1.12 0.47 1.59
finish show pivot 0.00429702 0.00 0.00 0.00 2.1059 1.12 0.47 1.59
start show pivot 0.000662088 0.00 0.00 0.00 2.10656 1.12 0.47 1.59
finish show pivot 0.00635386 0.01 0.00 0.01 2.11291 1.13 0.47 1.60
Finish 0.890132 0.36 0.03 0.39 3.00304 1.49 0.50 1.99

-- ThomasWeigert - 03 Dec 2006

Here for comparison the developer topic where DBCacheContrib wins huge:

without DBCacheContrib:

Mark Delta Absolute
time usr sys cpu time usr sys cpu
start xpShowDeveloperTimeSheet 0 0.00 0.00 0.00 0 0.00 0.00 0.00
start cache read 0.000423908 0.00 0.00 0.00 0.000423908 0.00 0.00 0.00
finish cache read 0.0251441 0.02 0.01 0.03 0.025568 0.02 0.01 0.03
finish xpShowDeveloperTimeSheet 5.7227 2.82 1.84 4.67 5.74827 2.84 1.85 4.70
start xpShowDeveloperEstimate 0.00151801 0.00 0.00 0.00 5.74978 2.84 1.85 4.70
finish xpShowDeveloperEstimate 3.81397 2.67 0.81 3.48 9.56375 5.52 2.66 8.18
start xpShowLoad 0.00131416 0.00 0.00 0.00 9.56507 5.52 2.66 8.18
finish xpShowLoad 4.24479 2.88 0.71 3.59 13.8099 8.40 3.37 11.78

with DBCacheContrib:

Mark Delta Absolute
time usr sys cpu time usr sys cpu
start xpShowDeveloperTimeSheet 0 0.00 0.00 0.00 0 0.00 0.00 0.00
start cache read 0.000429869 0.00 0.00 0.00 0.000429869 0.00 0.00 0.00
finish cache read 0.468249 0.40 0.06 0.46 0.468679 0.40 0.06 0.46
finish xpShowDeveloperTimeSheet 0.46266 0.45 0.00 0.45 0.931339 0.85 0.06 0.91
start xpShowDeveloperEstimate 0.00131083 0.00 0.00 0.00 0.93265 0.85 0.06 0.91
finish xpShowDeveloperEstimate 0.391639 0.38 0.00 0.38 1.32429 1.23 0.06 1.29
start xpShowLoad 0.00131822 0.00 0.00 0.00 1.32561 1.23 0.06 1.29
finish xpShowLoad 0.376841 0.34 0.00 0.34 1.70245 1.57 0.06 1.63

-- ThomasWeigert - 04 Dec 2006

Uploaded support for "managed caches", such that the application can control when to reload the database (obviously this can be much faster, if the application is smart about this).

-- ThomasWeigert - 04 Dec 2006

Bugs:Item4758 YetAnotherDBCacheContrib fails to create a subweb cache

Workaround Create for each web a folder in pub/_work_areas/DBCacheContrib/ with the appropriate owner/group (e.g. www-data).

-- FrankSpangenberg - 02 Oct 2007

Thomas, when you updated YetAnotherDBCacheContrib, the WYSIWYG editor changed a whole bunch of things. Did you change anything besides the TopicClassification from PluginPackage to ContribPackage?

-- PeterThoeny - 2009-07-12

Topic attachments
I Attachment History Action Size Date Who Comment
Texttxt warn_messages_ThomasFreudenberg_2006-11-10.txt r1 manage 12.0 K 2006-11-10 - 17:48 ThomasFreudenberg warning log with additional error info and installed plugin list
Edit | Attach | Watch | Print version | History: r12 < r11 < r10 < r9 < r8 | Backlinks | Raw View | Raw edit | More topic actions
Topic revision: r12 - 2009-07-12 - PeterThoeny
 
  • Learn about TWiki  
  • Download TWiki
This site is powered by the TWiki collaboration platform Powered by Perl Hosted by OICcam.com Ideas, requests, problems regarding TWiki? Send feedback. Ask community in the support forum.
Copyright © 1999-2017 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.