Tags:
benchmark1Add my vote for this tag development2Add my vote for this tag performance1Add my vote for this tag create new tag
, view all tags
Core code benchmarks for TWiki. Unzip the attached zip in a neutral directory.
  • Requires Storable (CPAN)
  • Requires a local installation of Athens
  • Requires the installations to be benchmarked to be at $server/version/ and $installdir/version, where version is the local name for that version e.g. "beta20040816" or "beijing"
  • Each installation has to have a 'benchmark' script installed in it's bin dir. This script is derived from the view script, and is therefore different for each major release. Appropriate benchmark scripts are at the top level of this zip; copy the right one to the bin directory of the installation and rename it 'benchmark'.
  • Requires a test page in the same place on all the installations; for standard benchmarks this should be WhatIsWikiWiki.
For examining performance of other functions of TWiki, then pick an appropriate page that exercises those aspects as much as possible. It is wise to stick to pages in TWiki web as far as possible, as they tend to have been around for a long time.

The benchmarks are run using the experiment script. This script runs the benchmark on Athens first (to normalise AthensMarks) and then repeats the experiment on a list of other installations. Edit the script as appropriate for the installations you want to benchmark.

Benchmark results are reported in a table that shows the plugins installed in the installation, together with an absolute number indicating the time taken to render the page, and the performance of the instalation in AthensMarks. AthensMarks are defined as the percentage of athens performance delivered by the installation; so 100 AthensMarks is the same performance as athens; 33 AthensMarks indicates performance 3 times worse than athens; 300 AthensMarks indicates performance 3 times better than athens.

The benchmarks are for the core code only i.e. the overhead of client/server comms is not taken into account. This means that factors such as time taken to download CSS or Javascript files are not being measured. For that, use ab based benchmarks.

Latest core benchmarks

Run by CrawfordCurrie - 24th October 2004

Release Skin Time per page AthensMarks
athens   .3266 100.0000
beijing   .3900 83.7400
cairo classic .4783 68.2800
MAIN classic .5450 59.9200
DEVELOP classic .4233 77.1500

Contributors: CrawfordCurrie, RafaelAlvarez, FrankHartmann

What happened on MAIN above? Nobody should be checking in any code that decreases performance by that much. The performance deterioration in Cairo is alrady hard to swallow, heaping even more on is really not good. Can we back these changes out or move the DEVELOP branch over?

-- ThomasWeigert - 25 Oct 2004

I honestly don't know; there have been no significant code changes. Here are some possible reasons, all of which probably contribute.

  1. a copy-paste of code for internationalisation that resulted in a second "do TWiki.cfg" in Render.pm.
  2. extractParameters
  3. increased code volume
  4. an artifact of perl, showing up in the order the benchmarks are run
  5. changes to the templates, resulting in greater processing requirement
DProf doesn't indicate any specifics. Try doing a svn diff -r 1742 in MAIN, and see if you can spot the guitly party.

Note that even after repeated runs, this result is consistent; MAIN is definitely 7-8 AthensMarks slower than Cairo.

IMHO we should not merge DEVELOP until a few more beady eyes have reviewed it. It may be faster because it isn't doing something.

-- CrawfordCurrie - 25 Oct 2004

Good point about the I18N cut-and-paste - the do "TWiki.cfg" in Render.pm is completely unnecessary, and the BEGIN block should look more like Search.pm, i.e.:

BEGIN {
    # Do a dynamic 'use locale' for this module
    if( $TWiki::useLocale ) {
        require locale;
        import locale ();
    }
}

The cost of this code should be quite low since it is just doing a use locale and the locale.pm module has already been require'd by TWiki.pm.

Now committed on MAIN in SVN.

-- RichardDonkin - 25 Oct 2004

I think I found a "bug" in the experiment.pl script. In the run sub, see the following code:


    my ( $code, $skin ) = @_;
    if ( $skin ) {
        $skin="?skin=$2";
    }

$2 is not defined in the context, so $skin will get the value ?skin=. This causes that most cairo-based distros will be measured using the pattern skin.

it should read:


    my ( $code, $skin ) = @_;
    if ( $skin ) {
        $skin="?skin=$skin";
    }

What's the difference?

With the original code these are the results (DEVELOP and trunk versions are 3183 and 1819):

TWiki core code benchmarks

Release Skin Plugins Time per page AthensMarks
base/athens   DefaultPlugin, InterwikiPlugin 0.561111111111111 100
base/beijing   DefaultPlugin, InterwikiPlugin 0.63125 88.8888888888889
base/cairo classic DefaultPlugin, InterwikiPlugin 1.036 54.1613041613042
base/trunk classic DefaultPlugin, InterwikiPlugin 0.923333333333333 60.7701564380265
base/DEVELOP classic DefaultPlugin, InterwikiPlugin 0.74 75.8258258258258

with the fix:

TWiki core code benchmarks

Release Skin Plugins Time per page AthensMarks
base/athens   DefaultPlugin, InterwikiPlugin 0.56 100
base/beijing   DefaultPlugin, InterwikiPlugin 0.622222222222222 90
base/cairo classic DefaultPlugin, InterwikiPlugin 0.951666666666667 58.8441330998249
base/trunk classic DefaultPlugin, InterwikiPlugin 0.926666666666667 60.431654676259
base/DEVELOP classic DefaultPlugin, InterwikiPlugin 0.715714285714286 78.2435129740519

-- RafaelAlvarez - 08 Nov 2004

Ach, sorry, Frank pointed that out before and my local version is correct already, it's just the uploaded package that was wrong. I have refreshed it.

-- CrawfordCurrie - 09 Nov 2004

Note: AthensMarks have been discredited/ignored by various authors and as a result I have stopped maintaining the scripts. Unfortunately no-one has come up with anything better. The TWiki::Monitor class offers a way to do detailed performance analysis, but there is no agreed TWiki benchmarking approach that can give a single simple number.

-- CrawfordCurrie - 28 Oct 2007

CategoryPerformance

Topic attachments
I Attachment History Action Size Date Who Comment
Compressed Zip archivezip BenchmarkContrib.zip r7 r6 r5 r4 r3 manage 13.5 K 2004-11-09 - 08:09 CrawfordCurrie With skin=$skin correction
Edit | Attach | Watch | Print version | History: r24 < r23 < r22 < r21 < r20 | Backlinks | Raw View | Raw edit | More topic actions
Topic revision: r24 - 2007-10-28 - CrawfordCurrie
 
  • Learn about TWiki  
  • Download TWiki
This site is powered by the TWiki collaboration platform Powered by Perl Hosted by OICcam.com Ideas, requests, problems regarding TWiki? Send feedback. Ask community in the support forum.
Copyright © 1999-2017 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.