search1Add my vote for this tag create new tag
, view all tags

Bug: SiteMap is slow

The new dynamically-created SiteMap is very nice, but somewhat slow - according to MartyBacke's comment of 4 Feb 2003 on FeedbackOnKnownIssuesOfTWiki01Feb2003, time to access WebHome goes from 3 to 6 seconds, which was almost bad enough for him to back out the Feb 2003 release. Similar comments on TwikiFreeBsdPerformance as well.

Since WebHome is commonly accessed, this creates an impression that TWiki is very slow even though it really isn't on most pages.

Test case

Access WebHome and compare speed to (say) WebStatistics.


Various including ModPerl - see pages linked above.

TWiki version: 01 Feb 2003
TWiki plugins:  
Server OS:  
Web server:  
Perl version:  
Client OS:  
Web Browser:  

-- RichardDonkin - 01 May 2003

Follow up

Not sure of the best fix here - the old SiteMap was harder to edit, I suppose, but much faster to access. Use of the CacheAddOn would solve this but is not a generally desirable approach.

  • ColasNahaboo : I think this shows the limit of the current approach of TWiki: too much runtime interpretation. I would encourage the CoreTeam to move towards more things being "compiled" when saving a page, to alleviate what is to be done when reading pages. A web page should definitely not load in more than 2 seconds (less than 0.5s is higly desirable). For instance, pre-compiling the sitemap as static data on each WebPreferences save.
    This is what I tried to achieve with KoalaSkin, see http://koala.ilog.fr/wiki/bin/oops/Koala?template=sitemap
    -- ColasNahaboo - 01 May 2003

Good point, some sort of 'save in compiled form' would be useful here. As a workaround, it's always possible to write a simpler hard-coded version of SiteMap and embed it into the WebHome pages - personally, I find the full SiteMap rather too large and complex for the intranet TWiki I use.

-- RichardDonkin - 02 May 2003

The problem is as follows: The SiteMap gets created dynamically, it lists all topics that have a SITEMAPLIST = on setting in the WebPreferences. The current TWiki does not allow to search for a topic name AND text within a topic, only one or the other. That is, the SiteMap currently searches all topics in all webs for a "\* Set SITEMAPLIST \= on" regex text, which is not scalable.

The correct solution is a SearchTopicNameAndTopicText feature to search only all WebPreferences topics for the SITEMAP setting. This will scale nicely, also if you have 100 webs.

-- PeterThoeny - 03 May 2003

In response to a user complaining of slow WebHome page loads ( TwikiFreeBsdPerformance ) I just browsed a few WebHome pages at twiki.org and was amazed at how slow they are now. If this is really due to the site maps, then I'd suggest we don't have site maps on the WebHome pages at all by default. Even if it wasn't slow, that information is really not necessary there IMHO - does anyone actually find it essential to have the full site map on each web's WebHome page? I think being able to see a list of the webs at the top is all you really need, and you can provide a link to a Site Map page for those who really want to see the details of other webs (like KoalaSkin does.)

Of course users can customise their individual install to remove the sitemap code from their WebHome pages, but when a new user starts they are likely to be rather tentative and reluctant to make a change like that until they understand their way around better. I know it took me a while to figure out which parts of the install were required and which were customizable, so I didn't remove anything from the default WebHome pages for a long time.

-- MartinWatt - 17 May 2003

Martin, that is an interesting observation. In fact, a few month ago we removed the sitemap at work from the WebHome topics except for the Main.WebHome topic. How about doing that with the TWiki distribution, and also add a SiteMap link to the top navigation?

-- PeterThoeny - 17 May 2003

I think that would be a good solution. Keeping the SiteMap on Main is reasonable as those users who do go there are often looking for more global information anyway. The main thing is to remove the SiteMap from the template used to create the new webs where users will spend most of their time.

-- MartinWatt - 17 May 2003

In MetaDataSucksMeasurement I point out that the way TWiki processes settings is extremely inefficient. One of the emasurements I made but didn't report there was of WebHome with the embedded SiteMap. For a 'regular' topic, the processing of settings takes about 25% of the time. Because of design problems, latency and repeated scanning, it takes about 90% of clock time and over 50% of CPU time to process SiteMap.

As I've pointed out in that thread, a number of performance enhancements are possible of the code in lib/TWiki/Prefs.pm is redesigned.

  1. "Get" the settings for each topic seperately and cache them
  2. Store the settings in a fast database outside of the topic body
  3. Separate access control from the other settings

The flow of control in processing SiteMap requires looking though the settings in each webs WebPreferences topic for three - and only those three - items. However scanning rsach of those topics is O(filesize), compared to a per-topic hash database which is independent of the size or number of settings, or a linear (e.g. CSV) per-topic store which is O(number of settings).

In addition, to scan that WebPreferences topic, its access has to be verified. The code that does this is highly redundant. With settings stored in the topic and access control as settings, caching the settings on a per topic basis would reduce the re-scanning. Even better would be storing access control outside the topic (the site and web preferences as well as the topic in question) in, for example, a DB_HASH for fast lookup.

I've been experimenting with this and the results are very positive.

In an ideal world, TWiki would be more OO and the storage could be done with a plug-in that over-rides the "base class method". Its all in the book.

I'm writing this up and will be posting it to Codev shortly. A "fix by design" rather than a "fix by patching the code".

-- AntonAylward - 17 May 2003

Fix record

The SearchTopicNameAndTopicText feature fixed this issue. For your own site, get the latest TWikiAlphaRelease and add a topic="%WEBPREFSTOPIC%" parameter to the formatted SEARCH in your TWiki.SiteMap.

-- PeterThoeny - 01 Nov 2003

I made some performance tuning: In search, in case topic parameter has no wildcards, that topic parameter is used for topic list instead of getting all topics first and limiting list to topic parameter. SiteMap page access time:

  • 12 sec, before topic="SiteMap" parameter
  •  4 sec, after adding topic="SiteMap" parameter
  •  2 sec, after performanc tuning

-- PeterThoeny - 02 Nov 2003

That's much nicer - even though TWiki.org is quite fast today, the sitemap here is really speedy. Thanks!

-- RichardDonkin - 06 Nov 2003

Edit | Attach | Watch | Print version | History: r15 < r14 < r13 < r12 < r11 | Backlinks | Raw View | Raw edit | More topic actions
Topic revision: r15 - 2004-08-20 - CrawfordCurrie
  • Learn about TWiki  
  • Download TWiki
This site is powered by the TWiki collaboration platform Powered by Perl Hosted by OICcam.com Ideas, requests, problems regarding TWiki? Send feedback. Ask community in the support forum.
Copyright © 1999-2015 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.