Tags:
create new tag
, view all tags

PluginBenchmarkAddOnDev Discussion: Page for developer collaboration, enhancement requests, patches and improved versions on PluginBenchmarkAddOn contributed by the TWikiCommunity.
• Please let us know what you think of this extension.
• For support, check the existing questions, or ask a new support question in the Support web!
• Please report bugs below

Feedback on the PluginBenchmarkAddOn

This is a first cut to make it easier to measure the PluginBenchmarks numbers. The DebugEnablePlugins patch needs to be applied.

Not sure of this add-on is ready for prime time. I experienced a big variation in performance numbers between runs of the pluginbenchmark utility.

-- PeterThoeny - 10 Nov 2004

Could you consider making a commandset for TWikiShellContrib for this addOn?

  • I first have to familiarize myself. Feel free to transmogrify this Add-on -- PTh

-- RafaelAlvarez - 10 Nov 2004

New version posted on Add-on topic:

  • More accurate benchmarks: Measure time of each topic 10 times, sort the result, use only the 5 fastest times. This excludes occational slow page access.
  • The utility can be configured for tools other then geturl, such as wget
  • Added doc recommending to run benchmarks only with CPAN:Time::HiRes

Please do not use the older version I posted yesterday, it is not accurate.

-- PeterThoeny - 10 Nov 2004

I am having trouble with this AddOnPackage... I installed it and it seems to work, but the results I get are not credible to me.

Consider the following two runs:

Plugin TWiki.FormattedSearch TWiki.GoodStyle Plugin topic
SpreadSheetPlugin 101% 102% 99%
EditTablePlugin 99% 100% 101%

These do not duplicate the results posted on PluginBenchmarkReport for these plugins, nor does it make sense to me. Any ideas what I might be doing wrong?

-- ThomasWeigert - 03 Mar 2005

Indeed that does not look right. Things to check:

  • Is mod_perl or SpeedyCGI disabled?
  • Do you run the utility on the web server?
  • Does your TWiki have TWiki:Codev/DebugEnablePlugins patch applied?
  • Is CPAN:Time::HiRes installed?
  • Increase the $repeats variable to spread out the tests

-- PeterThoeny - 03 Mar 2005

Answers:

  • Yes
  • Yes
  • Yes
  • Yes
  • Will try...

Do you think running the utility on the server is wrong?

-- ThomasWeigert - 03 Mar 2005

checked .zip into CVS

-- WillNorris - 19 Jul 2005

I dont seem to be getting consistent results using this AddOn. Is the version on the plugin topic up to date?

-- AndrewRJones - 05 Feb 2007

This add-on has not been tested with the latest TWiki releases, although it should work. Is Time::HiRes installed on your system?

-- PeterThoeny - 05 Feb 2007

Yes I have Time::HiRes installed, but still no luck.

-- AndrewRJones - 07 Feb 2007

I tried using this to benchmark the AutoCompletePlugin, using the version found in svn. There are the results I got after running it 3 times:

debian:/var/www/twiki2/tools# ./pluginbenchmark AutoCompletePlugin GoodStyle FormattedSearch Main.TWikiUsers
TWiki Plugin benchmark utility, version 11 Nov 2004. (Time::HiRes is installed and used)
Measuring, please be patient...
Topic:                          Without:  With:     Percent:
------------------------------  --------  --------  --------
TWiki.AutoCompletePlugin           0.036     0.037       98%
TWiki.GoodStyle                    0.036     0.036      101%
TWiki.FormattedSearch              0.036     0.036      100%
Main.TWikiUsers                    0.045     0.043      104%
debian:/var/www/twiki2/tools# ./pluginbenchmark AutoCompletePlugin GoodStyle FormattedSearch Main.TWikiUsers
TWiki Plugin benchmark utility, version 11 Nov 2004. (Time::HiRes is installed and used)
Measuring, please be patient...
Topic:                          Without:  With:     Percent:
------------------------------  --------  --------  --------
TWiki.AutoCompletePlugin           0.036     0.036       99%
TWiki.GoodStyle                    0.036     0.036       99%
TWiki.FormattedSearch              0.036     0.036       99%
Main.TWikiUsers                    0.035     0.036       98%
debian:/var/www/twiki2/tools# ./pluginbenchmark AutoCompletePlugin GoodStyle FormattedSearch Main.TWikiUsers
TWiki Plugin benchmark utility, version 11 Nov 2004. (Time::HiRes is installed and used)
Measuring, please be patient...
Topic:                          Without:  With:     Percent:
------------------------------  --------  --------  --------
TWiki.AutoCompletePlugin           0.036     0.036      101%
TWiki.GoodStyle                    0.058     0.050      116%
TWiki.FormattedSearch              0.039     0.048       82%
Main.TWikiUsers                    0.035     0.036       97%

Do these look right? Why am I getting >100%?

-- AndrewRJones - 25 Jul 2007

Sorry ignore that last comment. When I run ./geturl.pl localhost '/bin/view/Main/WebHome' I get a 403 Forbidden page. Not sure why though but its not being caused by this AddOn.

-- AndrewRJones - 25 Jul 2007

I have updated the addon. But I think the benefit of the statistics is quetionable. Since plugins use registerTagHandler, the impact on 'normal' topics like GoodStyle, FormattedSearch and Main.TWikiUsers is almost zero. The only topic that would influence the stats is the plugin topic itself, but only if the topic has plugin tags (not for instance a Sandbox example topic).

Currently classic skin is loaded, but it would make sense to use default skin.

-- ArthurClemens - 25 Oct 2008

It depends. Not all plugins use registerTagHandler, and some plugins spend time initializing the plugin. If another topic than the plugin topic itself has same variables of that plugin, than that other topic should be included in the benchmarks. The idea is to give the admin who considers installing a plugin some facts to decide if to install it or not.

Good question on what skin to use. The default PatternSkin represents real time use as perceived by users, whereas the ClassicSkin is much leaner, e.g. plugin performance issues will be more visible.

-- PeterThoeny - 25 Oct 2008

I meant to use skin=default, not pattern skin. That skin is even more bare than classic skin.

-- ArthurClemens - 25 Oct 2008

Yes, that is better. I never understood the need to create the ClassicSkin as an actual skin (vs. a pointer in SkinBrowser to the default skin).

-- PeterThoeny - 25 Oct 2008

Now with default skin.

But the numbers I get are not useful. For instance, TestCases.TestCaseTablePlugin is a long topic with a lot of tables, yet it renders at 100%.

./pluginbenchmark TablePlugin TestCases.TestCaseTablePlugin FormattedSearch Main.TWikiUsers

Topic:                          Without:  With:     Percent:
------------------------------  --------  --------  --------
TWiki.TablePlugin                  0.699     0.702      100%
TestCases.TestCaseTablePlugin      1.186     1.188      100%
TWiki.FormattedSearch              1.713     1.708      100%
Main.TWikiUsers                    0.561     0.557      100%

And FormPlugin has the examples on Sandbox.FormPluginExamples (the plugin topic has 1 example). Yet the example topic performs at 100%, and the plugin topic at 99%!

./pluginbenchmark FormPlugin Sandbox.FormPluginExamples FormattedSearch Main.TWikiUsers

Topic:                          Without:  With:     Percent:
------------------------------  --------  --------  --------
TWiki.FormPlugin                   0.758     0.764       99%
Sandbox.FormPluginExamples         0.718     0.721      100%
TWiki.FormattedSearch              1.732     1.715      100%
Main.TWikiUsers                    0.589     0.573      100%

-- ArthurClemens - 26 Oct 2008

Thanks Arthur for the TLC! The add-on has some deficiencies, errors are not reported properly for example. May be an auth problem? See AndrewRJones's note on 25 Jul 2007.

-- PeterThoeny - 26 Oct 2008

This extension does not reliably report performance numbers, it needs some TLC. I think the network & cgi env influence is too big, possibly better to run via shell script only.

For now I just did some doc fixes.

-- Peter Thoeny - 2016-01-17

Edit | Attach | Watch | Print version | History: r31 < r30 < r29 < r28 < r27 | Backlinks | Raw View | Raw edit | More topic actions
Topic revision: r31 - 2016-01-17 - PeterThoeny
 
  • Learn about TWiki  
  • Download TWiki
This site is powered by the TWiki collaboration platform Powered by Perl Hosted by OICcam.com Ideas, requests, problems regarding TWiki? Send feedback. Ask community in the support forum.
Copyright © 1999-2017 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.